OfF:6SSKrSSKrSSKrSSKrSSKJr SSKJrJrJ r SSK J r J r SSK JrJr SrSrSrS rS rS rS rS rSrSrSrSrSSjrSrSSjr\"SS/5r\"/SQ5r Sr!\"S:Xa\"5 gg)N)treebank)BrillTaggerTrainer RegexpTagger UnigramTagger)PosWord)Template error_listc[5 g)zx Run a demo with defaults. See source comments for details, or docstrings of any of the more specific demo_* functions. Npostag//usr/lib/python3/dist-packages/nltk/tbl/demo.pydemors  Hrc[SS9 g)F Exemplify repr(Rule) (see also str(Rule) and Rule.format("verbose")) repr ruleformatNr rrrdemo_repr_rule_formatrs  frc[SS9 g)rstrrNr rrrdemo_str_rule_formatr$s  erc[SS9 g)z" Exemplify Rule.format("verbose") verboserNr rrrdemo_verbose_rule_formatr+s  i rc@[[[/SQ55/S9 g)a The feature/s of a template takes a list of positions relative to the current word where the feature should be looked for, conceptually joined by logical OR. For instance, Pos([-1, 1]), given a value V, will hold whenever V is found one step to the left and/or one step to the right. For contiguous ranges, a 2-arg form giving inclusive end points can also be used: Pos(-3, -1) is the same as the arg below. ) templatesN)r r rrrrdemo_multiposition_featurer$2s hs<0123rc V[[[S/5[SS/55/S9 g)z0 Templates can have more than a single feature. rr r!r"N)r r rrrrrdemo_multifeature_templater&As$ htQCy#r2h-89:rc[SSS9 g)aP Show aggregate statistics per template. Little used templates are candidates for deletion, much used templates may possibly be refined. Deleting unused templates is mostly about saving time and/or space: training is basically O(T) in the number of templates T (also in terms of memory usage, which often will be the limiting factor). T)incremental_statstemplate_statsNr rrrdemo_template_statisticsr*Hs T$7rc[R"/SQSS/SS9n[R"/SQSS/SS9n[[R"X/SS 95n[ S R [U555 [USSS 9 g ) z Template.expand and Feature.expand are class methods facilitating generating large amounts of templates. See their documentation for details. Note: training with 500 templates can easily fill all available even on relatively small corpora )r!rr,F) excludezero)r r!rr,T)r,) combinationsz8Generated {} templates for transformation-based learning)r#r(r)N) rexpandrlistr printformatlenr )wordtplstagtplsr#s rdemo_generated_templatesr8Tss{{:1v5AHjj!QTBGX__h%8vNOI BII  N   Y$tLrc[SSSS9 g)zr Plot a learning curve -- the contribution on tagging accuracy of the individual rules. Note: requires matplotlib Tzlearningcurve.png)r(separate_baseline_datalearning_curve_outputNr rrrdemo_learning_curver<hs  #1rc[SS9 g)zO Writes a file with context for each erroneous word after tagging testing data z errors.txt) error_outputNr rrrdemo_error_analysisr?us  %rc[SS9 g)za Serializes the learned tagger to a file in pickle format; reloads it and validates the process. z tagger.pcl)serialize_outputNr rrrdemo_serialize_taggerrB|s  L)rc[SSSS9 g)z Discard rules with low accuracy. This may hurt performance a bit, but will often produce rules which are more interesting read to a human. i gQ? ) num_sentsmin_acc min_scoreNr rrrdemo_high_accuracy_rulesrHs  T426rc U=(d [nUcSSKJnJn U"5n[ XX(U5unnnnU(a[ R RU5(dP[UUS9n[US5n[R"UU5 SSS5 [SRU55 [U5n[R"U5n[SU35 SSS5 O[UUS9n[S5 U(a)[S RWRU555 [ R "5n[#WXU S 9n[S 5 UR%UX4U5n[S [ R "5U- S S35 U(a[SURU5-5 US:XaP[S5 ['UR)5S5H'unn[USSURU 5S35 M) U (a[S5 UR+UU5un n![S5 U(d [S5 UR-5n"U (aUR/U!5 U(a[1UU!U"US9 [SU35 O3[S5 UR3U5n U (aUR/5 U bs[U S5n#U#R5SU -5 U#R5SR7[9UU 55R;S5S-5 SSS5 [SU 35 U bUR3U5n [U S5n[R"UU5 SSS5 [SU 35 [U 5n[R"U5n$SSS5 [SU 35 UR3U5n%U U%:Xa [S 5 g[S!5 gg!,(df  GNr=f!,(df  GN=f!,(df  N=f!,(df  N=f!,(df  N=f)"a Brill Tagger Demonstration :param templates: how many sentences of training and testing data to use :type templates: list of Template :param tagged_data: maximum number of rule instances to create :type tagged_data: C{int} :param num_sents: how many sentences of training and testing data to use :type num_sents: C{int} :param max_rules: maximum number of rule instances to create :type max_rules: C{int} :param min_score: the minimum score for a rule in order for it to be considered :type min_score: C{int} :param min_acc: the minimum score for a rule in order for it to be considered :type min_acc: C{float} :param train: the fraction of the the corpus to be used for training (1=all) :type train: C{float} :param trace: the level of diagnostic tracing output to produce (0-4) :type trace: C{int} :param randomize: whether the training data should be a random subset of the corpus :type randomize: C{bool} :param ruleformat: rule output format, one of "str", "repr", "verbose" :type ruleformat: C{str} :param incremental_stats: if true, will tag incrementally and collect stats for each rule (rather slow) :type incremental_stats: C{bool} :param template_stats: if true, will print per-template statistics collected in training and (optionally) testing :type template_stats: C{bool} :param error_output: the file where errors will be saved :type error_output: C{string} :param serialize_output: the file where the learned tbl tagger will be saved :type serialize_output: C{string} :param learning_curve_output: filename of plot of learning curve(s) (train and also test, if available) :type learning_curve_output: C{string} :param learning_curve_take: how many rules plotted :type learning_curve_take: C{int} :param baseline_backoff_tagger: the file where rules will be saved :type baseline_backoff_tagger: tagger :param separate_baseline_data: use a fraction of the training data exclusively for training baseline :type separate_baseline_data: C{bool} :param cache_baseline_tagger: cache baseline tagger to this file (only interesting as a temporary workaround to get deterministic output from the baseline unigram tagger between python versions) :type cache_baseline_tagger: C{string} Note on separate_baseline_data: if True, reuse training data both for baseline and rule learner. This is fast and fine for a demo, but is likely to generalize worse on unseen data. Also cannot be sensibly used for learning curves on training data (the baseline will be artificially high). Nr)brill24describe_template_sets)backoffwz)Trained baseline tagger, pickled it to {}zReloaded pickled tagger from zTrained baseline taggerz! Accuracy on test set: {:0.4f}rzTraining tbl tagger...zTrained tbl tagger in z0.2fz secondsz Accuracy on test set: %.4fr,z Learned rules: 4d szJIncrementally tagging the test data, collecting individual rule statisticsz Rule statistics collectedzbWARNING: train_stats asked for separate_baseline_data=True; the baseline will be artificially high)takez Wrote plot of learning curve to zTagging the test datazErrors for Brill Tagger %r  zutf-8z)Wrote tagger errors including context to zWrote pickled tagger to z4Reloaded tagger tried on test set, results identicalz;PROBLEM: Reloaded tagger gave different results on test set) REGEXP_TAGGERnltk.tag.brillrJrK_demo_prepare_dataospathexistsropenpickledumpr3r4loadaccuracytimertrain enumeraterulesbatch_tag_incremental train_statsprint_template_statistics _demo_plot tag_sentswritejoinr encode)&r# tagged_datarE max_rulesrGrFr_trace randomizerr(r)r>rAr;learning_curve_takebaseline_backoff_taggerr:cache_baseline_taggerrJrK training_data baseline_data gold_data testing_databaseline_tagger print_rulestbrilltrainer brill_taggerrulenorule taggedtest teststats trainstatsfbrill_tagger_reloadedtaggedtest_reloadeds& rr r sp6FB I >PI2H?;]M9lww~~344+'>O+S1[ O[92 ;BB)  ' (K$kk+6O 12G1HI J) (( ?VW '( / 6 6((3  YY[F jG "#== gNL "499;#7"=X FG .1F1Fy1QQR z !"%l&8&8&:A>LFD VBKqZ!8 ;< =?  X #/"D"D )# Y -.% , "--/   2 29 = %y*CV  45J4KL M %&!++L9   2 2 4 , $ GG47GG H GGDIIjJ?@GGPSWW X% 9,HI#!++L9 "C (K KK k 2) ()9(:;< " #{$*KK $< !$ -.>-?@A*44\B , , H I O P$U21) (z% $) ($ #s=5O8:%P AP"P-P>8 P P P*- P;> Q c Uc [S5 [R"5nUb[U5U::a [U5nU(a5[R "[U55 [R "U5 [X!-5nUSUnXUnUVV s/sHoV s/sHoSPM sn PM n nn U(dUn O[U5S-n USU XlSpk[U5up[U 5unn[U 5unn[SUSSUSS35 [SU SSUSS35 [S RUUU(aS OS 55 XkXz4$s sn fs sn nf) Nz%Loading tagged data from treebank... rr/zRead testing data (dz sents/z wds)zRead training data (z-Read baseline data ({:d} sents/{:d} wds) {:s}z[reused the training set]) r3r tagged_sentsr5randomseedshuffleint corpus_sizer4)rjr_rErmr:cutoffrqrssenttrtrr bl_cutoff trainseqs traintokenstestseqs testtokens bltrainseqs bltraintokenss rrUrUQsq  56++- C , 9 $  C $%{# " #F(M9-I5>?YT4(4aqT4(YL? !%  &!+ *9 % * %& +=9Y(6Xz#.}#= [- |7:a. FG 1 W[O5 IJ 7>>  (B.I  ) BB+)?s E!E /EEcUS/nUSHnURUSU- 5 M USUVs/sH nSXaS- - PM nnUS/nUSHnURUSU- 5 M USUVs/sH nSXbS- - PM nnSSKJn [[ [ U555n UR XX5 UR/SQ5 URU5 gs snfs snf)N initialerrors rulescoresr!r, tokencountr)NNNg?) appendmatplotlib.pyplotpyplotr2ranger5plotaxissavefig) r;r}r~rQ testcurve rulescorex traincurvepltrs rrereys?+,I|, 223-:CET:JK:JQQ<000:JIK_-.J - *R.945. "#AHHQ1)HH $%KK%&L Os C'7C,z^-?[0-9]+(\.[0-9]+)?$CDz.*NN) r)z(The|the|A|a|An|an)$AT)z.*able$JJ)z.*ness$r)z.*ly$RB)z.*s$NNS)z.*ing$VBG)z.*ed$VBDrc<[U5[SU554$)Nc38# UHn[U5v M g7f)N)r5).0rs r corpus_size..s04a3q664s)r5sum)seqss rrrs Is0400 11r__main__)NNi,r/Ng?r/FrFFNNNrNFN)NN)#rVrZrr^ nltk.corpusrnltk.tagrrrrTrrnltk.tblr r rrrrr$r&r*r8r<r?rBrHr rUre NN_CD_TAGGERrSr__name__rrrrs DD$) ! 4; 8M( &*7    'BQJ%CP'&=}MN    2 zr