As no one answer to you (i think it because of the question itself, solution of which are easy, and have a lot of ways, and google can help with most of them), so to avoid silence in the forum, there is detailed answer:

1. any scripting language (python, perl, arexx all in all) in the few strings, like:

Perl

aos4shell:> perl -nle "$w{$_}++ for grep /\w/, map { s/[\. ,]*$//g; lc($_) } split; sub END { printf("%s\n",  $w) while (($w) = each(%w)) }" < input_file.txt >words.txt

You can of course make it as script.pl, and write it not in single line, but just as small script, like:

#!/usr/bin/perl
 
user strict;
 
my %count = ();
while (read(STDIN, $_, 4095) and $_ .= <STDIN>) {
    tr/A-Za-z/ /cs;
    ++$count{$_} foreach split(' ', lc $_);
}
 
my @lines = ();
my ($w, $c);
push(@lines, sprintf("%7d\t%s\n", $c, $w)) while (($w, $c) = each(%count));
print sort { $b cmp $a } @lines;

Running line will be:

aos4shell:> perl sort.pl < input.txt >output.txt

2. Simply script on bash with usage of unix's "sort" command line tool, like:

#!/usr/bin/bash
echo Enter the filename
read fn
 
 for WORD in $(cat $fn)
 do
  echo "$WORD "
 done | sdk:local/c/sort -u

Running will be just something like:

aos4shell:> sh sort.sh 

And then type file name with words which need to parse. Or just do redirecting like sh sort.sh >ready, and type filename as well. Btw, to be noted, don't mess aos4 version of "sort" binary which are in the system:c/ , with unix "sort" binary, which are placed in the sdk:local/c/.

3. Unix command line programms which aos4 have, like: sed, awk, gawk

Example on awk combined with sort:

aos4shell:> awk "{c=split($0, s); for(n=1; n<=c; ++n) print s[n] }" <file.txt | sdk:local/c/sort -u

4. As well as you can write the same on C or anything else.

ps. Blah, 2 days of silence and when i write post i realise that jaokim also answer in the same time :) But maybe my answer also can be helpfull.

You could do it easily in Gui4Cli too. (using any of the suggested tools or using xlistview events and the gui4Cli command set)
You then would have just as easily have a gui that you can expand upon. Working bottom up you can integrate new ideas and have a working gui all the time.

I am interested in the possibility to collect words too. As a basis for document search, selecting the most usefull tag words, spellchecking & translation, a lot of applications can use this.

Integrated in the OS, why not, as a background something working on the set of text files(directories) you indicate. Maybe an application for X1000's second processor?