OfSrSSKrSSKrSSKrSSKJr SSKJr SSK J r J r J r SSK JrJrJr SSKJrJrJr SSKJr SS KJr SS KJr S r"S S \5r\r"SS5r"SS\5r"SS\5r "SS\ 5r!"SS\ 5r""SS\5r#S(Sjr$Sr%Sr&S(Sjr'Sr(Sr)S)S jr*"S!S"\5r+S#r,S*S$jr-S%r.S&r/\0S':Xa\/"5 gg!\a Nf=f)+a A classifier model based on maximum entropy modeling framework. This framework considers all of the probability distributions that are empirically consistent with the training data; and chooses the distribution with the highest entropy. A probability distribution is "empirically consistent" with a set of training data if its estimated frequency with which a class and a feature vector value co-occur is equal to the actual frequency in the data. Terminology: 'feature' ====================== The term *feature* is usually used to refer to some property of an unlabeled token. For example, when performing word sense disambiguation, we might define a ``'prevword'`` feature whose value is the word preceding the target word. However, in the context of maxent modeling, the term *feature* is typically used to refer to a property of a "labeled" token. In order to prevent confusion, we will introduce two distinct terms to disambiguate these two different concepts: - An "input-feature" is a property of an unlabeled token. - A "joint-feature" is a property of a labeled token. In the rest of the ``nltk.classify`` module, the term "features" is used to refer to what we will call "input-features" in this module. In literature that describes and discusses maximum entropy models, input-features are typically called "contexts", and joint-features are simply referred to as "features". Converting Input-Features to Joint-Features ------------------------------------------- In maximum entropy models, joint-features are required to have numeric values. Typically, each input-feature ``input_feat`` is mapped to a set of joint-features of the form: | joint_feat(token, label) = { 1 if input_feat(token) == feat_val | { and label == some_label | { | { 0 otherwise For all values of ``feat_val`` and ``some_label``. This mapping is performed by classes that implement the ``MaxentFeatureEncodingI`` interface. N) defaultdict) ClassifierI) call_megamparse_megam_weightswrite_megam_file) call_tadmparse_tadm_weightswrite_tadm_file) CutoffCheckeraccuracylog_likelihood)gzip_open_unicode)DictionaryProbDist) OrderedDictz epytext enc\rSrSrSrSSjrSrSrSrSr Sr SS jr SS jr SS jr S r/S Qr\SSj5rSrg)MaxentClassifierNa A maximum entropy classifier (also known as a "conditional exponential classifier"). This classifier is parameterized by a set of "weights", which are used to combine the joint-features that are generated from a featureset by an "encoding". In particular, the encoding maps each ``(featureset, label)`` pair to a vector. The probability of each label is then computed using the following equation:: dotprod(weights, encode(fs,label)) prob(fs|label) = --------------------------------------------------- sum(dotprod(weights, encode(fs,l)) for l in labels) Where ``dotprod`` is the dot product:: dotprod(a,b) = sum(x*y for (x,y) in zip(a,b)) cfXlX lX0lUR5[ U5:Xdeg)a Construct a new maxent classifier model. Typically, new classifier models are created using the ``train()`` method. :type encoding: MaxentFeatureEncodingI :param encoding: An encoding that is used to convert the featuresets that are given to the ``classify`` method into joint-feature vectors, which are used by the maxent classifier model. :type weights: list of float :param weights: The feature weight vector for this classifier. :type logarithmic: bool :param logarithmic: If false, then use non-logarithmic weights. N) _encoding_weights _logarithmiclengthlen)selfencodingweights logarithmics 6/usr/lib/python3/dist-packages/nltk/classify/maxent.py__init__MaxentClassifier.__init__as-"" ' CL000c6URR5$N)rlabelsrs rr$MaxentClassifier.labelsxs~~$$&&r!cbXlURR5[U5:Xdeg)z Set the feature weight vector for this classifier. :param new_weights: The new feature weight vector. :type new_weights: list of float N)rrrr)r new_weightss r set_weightsMaxentClassifier.set_weights{s) $ ~~$$&#k*::::r!cUR$)zO :return: The feature weight vector for this classifier. :rtype: list of float )rr%s rrMaxentClassifier.weightss }}r!c@URU5R5$r#) prob_classifymax)r featuresets rclassifyMaxentClassifier.classifys!!*-1133r!c`0nURR5H{nURRX5nUR(a'SnUHupgXPRUU-- nM XRU'MVSnUHupgXRUU--nM XU'M} [ X RSS9$)Ng?T)log normalize)rr$encoderrr) rr0 prob_dictlabelfeature_vectortotalf_idf_valprods rr.MaxentClassifier.prob_classifys ^^**,E!^^22:EN  #1KD]]40588E$2#(% #1KDMM$/588D$2#'% - ")1B1BdSSr!c z^^ ^SnS[US- 5-S-nTRU5m [T R5T RSS9nUSUn[ SR U5S RS U55-5 [ S S US- S [U5----5 [[5m[U5HupgTRRX5nURU4SjSS9 UHupTR(aTR U U -n OTR U U -n TRR#U 5n U R%S5Sn U SU -- n [U 5S:aU SSS-n [ XLUS -S-U 4-5 TU==U - ss'M M [ S S US- S [U5----5 [ SR U5S RU4SjU55-5 [ SR U5S RU 4SjU55-5 g)z Print a table showing the effect of each of the features in the given feature set, and how they combine to determine the probabilities of each label for that featureset. 2z %-zs%s%8.3fTkeyreverseNz Featurec38# UHnSSU-SS-v M g7f)z%8s%sN).0ls r +MaxentClassifier.explain..s?1eq"1~.sz -c:>[TRUS5$)Nrabsr)fid__rs r*MaxentClassifier.explain..s#dmmE!H&=">r! and label is rz (%s)/,z... z TOTAL:c34># UH nSTU-v M g7fz%8.3fNrJ)rKrLsumss rrMrNs3Vv!Gd1g4Evsz PROBS:c3L># UHnSTRU5-v M g7fr])prob)rKrLpdists rrMrNs>v!g 1 -vs!$)strr.sortedsamplesr`printljustjoinrrint enumeraterr7sortrrdescribesplit)rr0columns descr_widthTEMPLATEr$ir9r:r<r=scoredescrrar^s` @@rexplainMaxentClassifier.explains$  C a00:="":. UZZF!   k *gg??? @  dSK!Oa#f+o=>>?3!&)HA!^^22:EN   >   . $$ MM$/%7E MM$/58E//5 $45a85(u:?!#2J.EhQe!<<=U u$  . *" dSK!Oa#f+o=>>?   [ )BGG3Vv3V,V V     [ )gg>v>> ? r!c ^[TS5(aTRSU$[[[ [ TR 555U4SjSS9TlTRSU$)zG Generates the ranked list of informative features from most to least. _most_informative_featuresNc4>[TRU5$r#rR)fidrs rrU.sDMM#$6 7r!TrC)hasattrrvrclistrangerr)rns` rmost_informative_features*MaxentClassifier.most_informative_featuressd 45 6 6222A6 6.4U3t}}-./7/D + 222A6 6r!crURS5nUS:Xa'UVs/sHo@RUS:dMUPM nnO,US:Xa&UVs/sHo@RUS:dMUPM nnUSUH:n[URUSSURR U535 M< gs snfs snf)z :param show: all, neg, or pos (for negative-only or positive-only) :type show: str :param n: The no. of top features :type n: int Nposrnegz8.3frZ)r~rrerrk)rr}showfidsrxs rshow_most_informative_features/MaxentClassifier.show_most_informative_featuress--d3 5=#'B4C==+=+AC4DBD U]#'B4C==+=+AC4DB8C T]]3'-Qt~~/F/Fs/K.LM NCBsB/B/ B4#B4cS[URR55URR54-$)Nz:)rrr$rr%s r__repr__MaxentClassifier.__repr__s:K %%' ( NN ! ! #O   r!)GISIISMEGAMTADMNc TUcSnUHnUS;dM [SU-5e UR5nUS:Xa [XXE40UD6$US:Xa [XXE40UD6$US:Xa[ XXEU40UD6$US:Xa)Un X9S'XIS'XYS 'XiS '[ R "U40U D6$[S U-5e) aV Train a new maxent classifier based on the given corpus of training samples. This classifier will have its weights chosen to maximize entropy while remaining empirically consistent with the training corpus. :rtype: MaxentClassifier :return: The new maxent classifier :type train_toks: list :param train_toks: Training data, represented as a list of pairs, the first member of which is a featureset, and the second of which is a classification label. :type algorithm: str :param algorithm: A case-insensitive string, specifying which algorithm should be used to train the classifier. The following algorithms are currently available. - Iterative Scaling Methods: Generalized Iterative Scaling (``'GIS'``), Improved Iterative Scaling (``'IIS'``) - External Libraries (requiring megam): LM-BFGS algorithm, with training performed by Megam (``'megam'``) The default algorithm is ``'IIS'``. :type trace: int :param trace: The level of diagnostic tracing output to produce. Higher values produce more verbose output. :type encoding: MaxentFeatureEncodingI :param encoding: A feature encoding, used to convert featuresets into feature vectors. If none is specified, then a ``BinaryMaxentFeatureEncoding`` will be built based on the features that are attested in the training corpus. :type labels: list(str) :param labels: The set of possible labels. If none is given, then the set of all labels attested in the training data will be used instead. :param gaussian_prior_sigma: The sigma value for a gaussian prior on model weights. Currently, this is supported by ``megam``. For other algorithms, its value is ignored. :param cutoffs: Arguments specifying various conditions under which the training should be halted. (Some of the cutoff conditions are not supported by some algorithms.) - ``max_iter=v``: Terminate after ``v`` iterations. - ``min_ll=v``: Terminate after the negative average log-likelihood drops under ``v``. - ``min_lldelta=v``: Terminate if a single iteration improves log likelihood by less than ``v``. iis) max_itermin_ll min_lldeltamax_acc min_accdelta count_cutoffnormexplicit bernoullizUnexpected keyword arg %rgismegamtadmtracerr$gaussian_prior_sigmazUnknown algorithm %s) TypeErrorlower train_maxent_classifier_with_iis train_maxent_classifier_with_gis"train_maxent_classifier_with_megamTadmMaxentClassifiertrain ValueError) cls train_toks algorithmrrr$rcutoffsrDkwargss rrMaxentClassifier.trains |  IC   ;c ABBOO%  387> % 387> ' !585IMT & F#7O!): %8 -A) *'--jCFC C3i?@ @r!)rrrvr)T)) )rall)NNNr)__name__ __module__ __qualname____firstlineno____doc__rr$r)rr1r.rsr~rr ALGORITHMS classmethodr__static_attributes__rJr!rrrNsj$1.';4T(* X 7O  1JaAaAr!rc6\rSrSrSrSrSrSrSrSr Sr g ) MaxentFeatureEncodingIidap A mapping that converts a set of input-feature values to a vector of joint-feature values, given a label. This conversion is necessary to translate featuresets into a format that can be used by maximum entropy models. The set of joint-features used by a given encoding is fixed, and each index in the generated joint-feature vectors corresponds to a single joint-feature. The length of the generated joint-feature vectors is therefore constant (for a given encoding). Because the joint-feature vectors generated by ``MaxentFeatureEncodingI`` are typically very sparse, they are represented as a list of ``(index, value)`` tuples, specifying the value of each non-zero joint-feature. Feature encodings are generally created using the ``train()`` method, which generates an appropriate encoding based on the input-feature values and labels that are present in a given corpus. c[5e)a  Given a (featureset, label) pair, return the corresponding vector of joint-feature values. This vector is represented as a list of ``(index, value)`` tuples, specifying the value of each non-zero joint-feature. :type featureset: dict :rtype: list(tuple(int, int)) NotImplementedErrorrr0r9s rr7MaxentFeatureEncodingI.encode{ "##r!c[5e)zr :return: The size of the fixed-length joint-feature vectors that are generated by this encoding. :rtype: int rr%s rrMaxentFeatureEncodingI.length "##r!c[5e)z :return: A list of the "known labels" -- i.e., all labels ``l`` such that ``self.encode(fs,l)`` can be a nonzero joint-feature vector for some value of ``fs``. :rtype: list rr%s rr$MaxentFeatureEncodingI.labelss "##r!c[5e)z :return: A string describing the value of the joint-feature whose index in the generated feature vectors is ``fid``. :rtype: str rrrxs rrkMaxentFeatureEncodingI.describerr!c[5e)a7 Construct and return new feature encoding, based on a given training corpus ``train_toks``. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. r)rrs rrMaxentFeatureEncodingI.trainrr!rJN) rrrrrr7rr$rkrrrJr!rrrds , $$$$ $r!rc6\rSrSrSrSrSrSrSrSr Sr g ) #FunctionBackedMaxentFeatureEncodingiz A feature encoding that calls a user-supplied function to map a given featureset/label pair to a sparse joint-feature vector. c(X lXlX0lg)a Construct a new feature encoding based on the given function. :type func: (callable) :param func: A function that takes two arguments, a featureset and a label, and returns the sparse joint feature vector that encodes them:: func(featureset, label) -> feature_vector This sparse joint feature vector (``feature_vector``) is a list of ``(index,value)`` tuples. :type length: int :param length: The size of the fixed-length joint-feature vectors that are generated by this encoding. :type labels: list :param labels: A list of the "known labels" for this encoding -- i.e., all labels ``l`` such that ``self.encode(fs,l)`` can be a nonzero joint-feature vector for some value of ``fs``. N)_length_func_labels)rfuncrr$s rr,FunctionBackedMaxentFeatureEncoding.__init__s0   r!c$URX5$r#)rrs rr7*FunctionBackedMaxentFeatureEncoding.encodeszz*,,r!cUR$r#rr%s rr*FunctionBackedMaxentFeatureEncoding.length ||r!cUR$r#rr%s rr$*FunctionBackedMaxentFeatureEncoding.labelsrr!cg)Nzno description availablerJrs rrk,FunctionBackedMaxentFeatureEncoding.describes)r!)rrrN) rrrrrrr7rr$rkrrJr!rrrs  8-*r!rcN\rSrSrSrS SjrSrSrSrSr \ S S j5r S r g) BinaryMaxentFeatureEncodingia A feature encoding that generates vectors containing a binary joint-features of the form: | joint_feat(fs, l) = { 1 if (fs[fname] == fval) and (l == label) | { | { 0 otherwise Where ``fname`` is the name of an input-feature, ``fval`` is a value for that input-feature, and ``label`` is a label. Typically, these features are constructed based on a training corpus, using the ``train()`` method. This method will create one feature for each combination of ``fname``, ``fval``, and ``label`` that occurs at least once in the training corpus. The ``unseen_features`` parameter can be used to add "unseen-value features", which are used whenever an input feature has a value that was not encountered in the training corpus. These features have the form: | joint_feat(fs, l) = { 1 if is_unseen(fname, fs[fname]) | { and l == label | { | { 0 otherwise Where ``is_unseen(fname, fval)`` is true if the encoding does not contain any joint features that are true when ``fs[fname]==fval``. The ``alwayson_features`` parameter can be used to add "always-on features", which have the form:: | joint_feat(fs, l) = { 1 if (l == label) | { | { 0 otherwise These always-on features allow the maxent model to directly model the prior probabilities of each label. c[UR55[[[U555:wa [ S5e[ U5UlX l[U5UlSUl SUl U(aZ[U5VVs0sHupVXeUR-_M snnUl U=R[UR5- slU(aiUVVVs1sHupxogiM n nnn[U 5VVs0sHupWXuUR-_M snnUl U=R[U 5- slggs snnfs snnnfs snnf)a[ :param labels: A list of the "known labels" for this encoding. :param mapping: A dictionary mapping from ``(fname,fval,label)`` tuples to corresponding joint-feature indexes. These indexes must be the set of integers from 0...len(mapping). If ``mapping[fname,fval,label]=id``, then ``self.encode(..., fname:fval, ..., label)[id]`` is 1; otherwise, it is 0. :param unseen_features: If true, then include unseen value features in the generated joint-feature vectors. :param alwayson_features: If true, then include always-on features in the generated joint-feature vectors. HMapping values must be exactly the set of integers from 0...len(mapping)N setvaluesr|rrr{r_mappingr _alwayson_unseenri rr$mappingunseen_featuresalwayson_featuresrpr9fnamefvalfnamess rr$BinaryMaxentFeatureEncoding.__init__*" w~~ Cc'l(;$< <8  F| ( 97| <, , :CF:K:KJQ4<<'':KDN LLC/ /L 8?@ 4eeF@FOPVFWXFW Et||#33FWXDL LLCK 'L   AXE+E Ec/nUR5HupEXEU4UR;a$URURXEU4S45 M;UR(dMNURHnXEU4UR;dM Mt X@R;dMURURUS45 M UR (a/X R ;a URUR US45 U$Nr[)itemsrappendrrrrr0r9rrrlabel2s rr7"BinaryMaxentFeatureEncoding.encode6s&++-KEU#t}}4uE/A!BA FG"llFV, =+  , e)>e~~5 OOT^^E2A6 7r!c[U[5(d [S5eUR U[ UR 5:aURUupEnUSU<SU<3$UR(aMXRR5;a0URR5HupgX:XdM SU-s $ gUR(aMXRR5;a0URR5HupGX:XdM SU-s $ g[S5e![aS S/[ UR 5-UlUR R5Hup#X RU'M GNZf=fNzdescribe() expected an intz==rWz label is %rz %s is unseenzBad feature id isinstancerhr _inv_mappingAttributeErrorrrrrrrrrr<inforprrr9f_id2s rrk$BinaryMaxentFeatureEncoding.describeQL$$$89 9 ,    #dmm$ $#'#4#4T#: U%WBthnUI> > ^^(=(=(? ? $ 4 4 6 =(500!7\\dll&9&9&;; $ 2 2 4 =)E11!5-. .# ,!#s4=='9 9D ==..0'+!!!$1 , D,,AF F cUR$r#rr%s rr$"BinaryMaxentFeatureEncoding.labelsj ||r!cUR$r#rr%s rr"BinaryMaxentFeatureEncoding.lengthnrr!Nc X0n[5n[[5nUH|upU(aX;a[SU -5eUR U 5 UR 5H8upXzU 4==S- ss'XzU 4U:dMXU 4U;dM)[ U5XZX4'M: M~ UcUnU"X540UD6$)a Construct and return new feature encoding, based on a given training corpus ``train_toks``. See the class description ``BinaryMaxentFeatureEncoding`` for a description of the joint-features that will be included in this encoding. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. :type count_cutoff: int :param count_cutoff: A cutoff value that is used to discard rare joint-features. If a joint-feature's value is 1 fewer than ``count_cutoff`` times in the training corpus, then that joint-feature is not included in the generated encoding. :type labels: list :param labels: A list of labels that should be used by the classifier. If not specified, then the set of labels attested in ``train_toks`` will be used. :param options: Extra parameters for the constructor, such as ``unseen_features`` and ``alwayson_features``. Unexpected label %sr[)rrrhraddrr rrrr$optionsr seen_labelscounttokr9rrs rr!BinaryMaxentFeatureEncoding.trainrs8e C $JC%- !6!>?? OOE " #yy{ Tk"a'"%5U+7:69'lt 23 + % > F6.g..r!rrrrrrFFrN rrrrrrr7rkr$rrrrrJr!rrrs6&P/(b6/20/0/r!rcF\rSrSrSrS Sjr\S5rSrSr Sr S r g) GISEncodingia A binary feature encoding which adds one new joint-feature to the joint-features defined by ``BinaryMaxentFeatureEncoding``: a correction feature, whose value is chosen to ensure that the sparse vector always sums to a constant non-negative number. This new feature is used to ensure two preconditions for the GIS training algorithm: - At least one feature vector index must be nonzero for every token. - The feature vector must sum to a constant non-negative number for every token. Nc [RXX#U5 Uc%[UVVVs1sHupgoiM snnn5S-nXPlgs snnnf)z :param C: The correction constant. The value of the correction feature is based on this value. In particular, its value is ``C - sum([v for (f,v) in encoding])``. :seealso: ``BinaryMaxentFeatureEncoding.__init__`` Nr[)rrr_C) rr$rrrCrrr9s rrGISEncoding.__init__sO $,, '4E  9w?w3UUw?@1DA@sA cUR$)zGThe non-negative constant that all encoded feature vectors will sum to.rr%s rr GISEncoding.Cs wwr!c[RXU5n[RU5n[SU55nXPR:a [ S5eUR X@RU- 45 U$)Nc3*# UH upUv M g7fr#rJ)rKfvs rrM%GISEncoding.encode..s-H&1AHz&Correction feature is not high enough!)rr7rsumrrr)rr0r9r base_lengthr;s rr7GISEncoding.encodesj.55dN188> -H-- GG EF Fggo67r!c2[RU5S-$r)rrr%s rrGISEncoding.lengths*11$7!;;r!c|U[RU5:XaSUR-$[RX5$)NzCorrection feature (%s))rrrrk)rr<s rrkGISEncoding.describes6 .55d; ;,tww6 6.77C Cr!r)FFN) rrrrrrpropertyrr7rrkrrJr!rrrs7 RV  <Dr!rcJ\rSrSrS SjrSrSrSrSr\ S Sj5r S r g) TadmEventMaxentFeatureEncodingic[U5Ul[5Ul[R XURX45 gr#)rr_label_mappingrr)rr$rrrs rr'TadmEventMaxentFeatureEncoding.__init__s1#G, )m#,, $-- r!c/nUR5HupEXB4UR;a#[UR5URXB4'XPR;aF[ U[ 5(d#[UR5URU'OXPRU'UR URXB4URU45 M U$r#)rrrr1rrhr)rr0r9rfeaturevalues rr7%TadmEventMaxentFeatureEncoding.encodes(..0NGt}}425dmm2D w.////!%--14T5H5H1ID''.16''. OO/0$2E2Ee2LM 1r!cUR$r#rr%s rr$%TadmEventMaxentFeatureEncoding.labelsrr!c`URHup#URX#4U:XdMX#4s $ gr#)r)rrxr4r9s rrk'TadmEventMaxentFeatureEncoding.describes/"mmNG}}g-.#5'',r!c,[UR5$r#)rrr%s rr%TadmEventMaxentFeatureEncoding.lengths4==!!r!Nc [5nU(d/n[U5nUHupgXs;dM URU5 M UH/upgUH$nUHnX4U;dM [U5XXU4'M M& M1 U"X540UD6$r#)rr{rr) rrrr$rrr0r9r4s rr$TadmEventMaxentFeatureEncoding.trains-F*% !+ J" e$",", J)G'w647L% 01 * ", 6.g..r!)r1rrr) rrrrrr7r$rkrrrrrJr!rr/r/s/  ( "//r!r/cN\rSrSrSrS SjrSrSrSrSr \ S S j5r S r g) TypedMaxentFeatureEncodingia A feature encoding that generates vectors containing integer, float and binary joint-features of the form: Binary (for string and boolean features): | joint_feat(fs, l) = { 1 if (fs[fname] == fval) and (l == label) | { | { 0 otherwise Value (for integer and float features): | joint_feat(fs, l) = { fval if (fs[fname] == type(fval)) | { and (l == label) | { | { not encoded otherwise Where ``fname`` is the name of an input-feature, ``fval`` is a value for that input-feature, and ``label`` is a label. Typically, these features are constructed based on a training corpus, using the ``train()`` method. For string and boolean features [type(fval) not in (int, float)] this method will create one feature for each combination of ``fname``, ``fval``, and ``label`` that occurs at least once in the training corpus. For integer and float features [type(fval) in (int, float)] this method will create one feature for each combination of ``fname`` and ``label`` that occurs at least once in the training corpus. For binary features the ``unseen_features`` parameter can be used to add "unseen-value features", which are used whenever an input feature has a value that was not encountered in the training corpus. These features have the form: | joint_feat(fs, l) = { 1 if is_unseen(fname, fs[fname]) | { and l == label | { | { 0 otherwise Where ``is_unseen(fname, fval)`` is true if the encoding does not contain any joint features that are true when ``fs[fname]==fval``. The ``alwayson_features`` parameter can be used to add "always-on features", which have the form: | joint_feat(fs, l) = { 1 if (l == label) | { | { 0 otherwise These always-on features allow the maxent model to directly model the prior probabilities of each label. c[UR55[[[U555:wa [ S5e[ U5UlX l[U5UlSUl SUl U(aZ[U5VVs0sHupVXeUR-_M snnUl U=R[UR5- slU(aiUVVVs1sHupxogiM n nnn[U 5VVs0sHupWXuUR-_M snnUl U=R[U 5- slggs snnfs snnnfs snnf)a] :param labels: A list of the "known labels" for this encoding. :param mapping: A dictionary mapping from ``(fname,fval,label)`` tuples to corresponding joint-feature indexes. These indexes must be the set of integers from 0...len(mapping). If ``mapping[fname,fval,label]=id``, then ``self.encode({..., fname:fval, ...``, label)[id]} is 1; otherwise, it is 0. :param unseen_features: If true, then include unseen value features in the generated joint-feature vectors. :param alwayson_features: If true, then include always-on features in the generated joint-feature vectors. rNrrs rr#TypedMaxentFeatureEncoding.__init__Trrc/nUR5GHupE[U[[45(aLU[ U5U4UR ;a.UR UR U[ U5U4U45 MkMmXEU4UR ;a$UR UR XEU4S45 MUR(dMURHnXEU4UR ;dM M X@R;dMUR URUS45 GM UR(a/X R;a UR URUS45 U$r) rrrhfloattyperrrrrrs rr7!TypedMaxentFeatureEncoding.encodes&&++-KE$e --4:u->OOT]]5$t*e3K%Ld$ST?'4==8OOT]]53E%F$JK\\\"&,,!0DMMA!#/ !LL0$OOT\\%-@!,DE'., >>e~~5 OOT^^E2A6 7r!c[U[5(d [S5eUR U[ UR 5:aURUupEnUSU<SU<3$UR(aMXRR5;a0URR5HupgX:XdM SU-s $ gUR(aMXRR5;a0URR5HupGX:XdM SU-s $ g[S5e![aS S/[ UR 5-UlUR R5Hup#X RU'M GNZf=frrrs rrk#TypedMaxentFeatureEncoding.describerrcUR$r#rr%s rr$!TypedMaxentFeatureEncoding.labelsrr!cUR$r#rr%s rr!TypedMaxentFeatureEncoding.lengthrr!Nc 0n[5n[[5nUHupU(aX;a[SU -5eUR U 5 UR 5H\up[ U 5[[4;a [ U 5n XzU 4==S- ss'XzU 4U:dMCXU 4U;dMM[U5XZX4'M^ M UcUnU"X540UD6$)aq Construct and return new feature encoding, based on a given training corpus ``train_toks``. See the class description ``TypedMaxentFeatureEncoding`` for a description of the joint-features that will be included in this encoding. Note: recognized feature values types are (int, float), over types are interpreted as regular binary features. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. :type count_cutoff: int :param count_cutoff: A cutoff value that is used to discard rare joint-features. If a joint-feature's value is 1 fewer than ``count_cutoff`` times in the training corpus, then that joint-feature is not included in the generated encoding. :type labels: list :param labels: A list of labels that should be used by the classifier. If not specified, then the set of labels attested in ``train_toks`` will be used. :param options: Extra parameters for the constructor, such as ``unseen_features`` and ``alwayson_features``. r r[) rrrhrr rrErDrr s rr TypedMaxentFeatureEncoding.trains>e C $JC%- !6!>?? OOE " #yy{ :#u-:DTk"a'"%5U+7:69'lt 23 + %" > F6.g..r!rrrrrJr!rr@r@s76p/(b@/25/5/r!r@c URSS5 [U5nUc[RXS9n[ US5(d [ S5eSUR - n[X5n[[R"US:H5S5n[R"[U5S5n UHn [RX'M [X)5n [R"U5n AUS:a[!S US-5 US :a [!5 [!S 5 [!S 5 US :aXUR"=(d [%X5n UR&=(d [)X5nUR*n[!S XU4-5 [-XU5nUHn UU ==S- ss'M [R"U5nAU R/5n XU- U-- n U R1U 5 UR3X5(aOMUS :a)[%X5n [)X5n[!SU SSUS35 U $![4a [!S5 NH e=f)aj Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the Generalized Iterative Scaling algorithm. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. rdr$rzJThe GIS algorithm requires an encoding that defines C (e.g., GISEncoding).r4rd ==> Training (%d iterations)rB- Iteration Log Likelihood Accuracy- --------------------------------------- %9d %14.5f %9.3fr[* Training stopped: keyboard interrupt Final 14.5f 9.3f) setdefaultr rrrzrrcalculate_empirical_fcountrnumpynonzerozerosrNINF ConditionalExponentialClassifierlog2rellr accr itercalculate_estimated_fcountrr)checkKeyboardInterrupt)rrrr$r cutoffcheckerCinvempirical_fcount unattestedrrx classifierlog_empirical_fcountrdreiternumestimated_fcountlog_estimated_fcounts rrrsZ z3'!'*M$$Z$? 8S ! ! -    D2*GU]]#3q#89!<=Jkk#./5Gzz 1(DJ!::&67 qy .1DDE qy  => => qy"%%O )O#''K8J+K',,3wC6HHI :   " %*%"#(::.>#?  !((*G /CCtK KG  " "7 +"":::5B qy J 3z. "2e*DT ;<  < :; s*CH7H77IIc[R"UR5S5nUH+up4URX45HupVX%==U- ss'M M- U$NrR)r^r`rr7)rrfcountrr9indexvals rr]r]fsM [[*C 0F "//#5JE MS M6! Mr!c([R"UR5S5nUHfupEURU5nUR 5H<nUR U5nUR XE5HupX8==Xy-- ss'M M> Mh U$rt)r^r`rr.rdr`r7) rnrrrurr9rar`rxrs rrgrgps} [[*C 0F ((-]]_E::e$D%__S8  t{* 9%! Mr!c URSS5 [U5nUc[RXS9n[ X5[ U5- n[ X5n[R"[XwRS9S5n[R"U[ U5S45n [[R"US:H5S5n [R"[ U5S5n U Hn [RX'M [!X+5n US:a[#SUS-5 US :a [#5 [#S 5 [#S 5 US :aXUR$=(d ['X5nUR(=(d [+X5nUR,n[#S UX4-5 [/UU U UUUU U5nU R15n U U- n U R3U 5 UR5X5(aOMUS :a)['X5n[+X5n[#SUSSUS35 U $![6a [#S 5 NH e=f)af Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the Improved Iterative Scaling algorithm. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. rrPrQ)rDrRr[rrSrBrTrUrVrWrXrYrZr[)r\r rrr]rcalculate_nfmapr^arrayrc __getitem__reshaperr_r`rarbrerdr rer rfcalculate_deltasrr)rhri)rrrr$rrjempirical_ffreqnfmapnfarray nftransposermrrxrnrdrerpdeltass rrrs6 z3'!'*M.44Z4O1FZXO J 1Ekk&,=,=>DG--#g,):;KU]]?a#78; => qy"%%O )O#''K8J+K',,3w6HHI& F!((*G v G  " "7 +"":::5B qy J 3z. "2e*DT ;<  < :; s B-H*7H**IIc [5nUHLup4UR5H3nUR[SUR X55555 M5 MN [ U5VVs0sHupgXv_M snn$s snnf)a Construct a map that can be used to compress ``nf`` (which is typically sparse). *nf(feature_vector)* is the sum of the feature values for *feature_vector*. This represents the number of features that are active for a given labeled text. This method finds all values of *nf(t)* that are attested for at least one token in the given list of training tokens; and constructs a dictionary mapping these attested values to a continuous range *0...N*. For example, if the only values of *nf()* that were attested were 3, 5, and 7, then ``_nfmap`` might return the dictionary ``{3:0, 5:1, 7:2}``. :return: A map that can be used to compress ``nf`` to a dense vector. :rtype: dict(int -> int) c3*# UH upUv M g7fr#rJrKidrws rrM"calculate_nfmap..sK/J)2#/Jr%)rr$r r&r7ri)rrnfsetr_r9rpnfs rrzrzsm* EE__&E IIcKxs/JKK L'"+5!1 2!1gqBE!1 22 2s+A=c |SnSn [R"UR5S5n [R"[ U5UR54S5n UHupUR U 5nUR 5HUn URX5n[SU55nUH)unnXUU4==URU 5U-- ss'M+ MW M U [ U5-n [U 5Hn[R"XZ5nSU-nUU-n[R"UU -SS9n[R"UU -SS9nUHnUU==S- ss'M XU- U*- -n [R"[UU- 55[R"[U 55- nUU:dMU s $ U $) aW Calculate the update values for the classifier weights for this iteration of IIS. These update weights are the value of ``delta`` that solves the equation:: ffreq_empirical[i] = SUM[fs,l] (classifier.prob_classify(fs).prob(l) * feature_vector(fs,l)[i] * exp(delta[i] * nf(feature_vector(fs,l)))) Where: - *(fs,l)* is a (featureset, label) tuple from ``train_toks`` - *feature_vector(fs,l)* = ``encoding.encode(fs,l)`` - *nf(vector)* = ``sum([val for (id,val) in vector])`` This method uses Newton's method to solve this equation for *delta[i]*. In particular, it starts with a guess of ``delta[i]`` = 1; and iteratively updates ``delta`` with: | delta[i] -= (ffreq_empirical[i] - sum1[i])/(-sum2[i]) until convergence, where *sum1* and *sum2* are defined as: | sum1[i](delta) = SUM[fs,l] f[i](fs,l,delta) | sum2[i](delta) = SUM[fs,l] (f[i](fs,l,delta).nf(feature_vector(fs,l))) | f[i](fs,l,delta) = (classifier.prob_classify(fs).prob(l) . | feature_vector(fs,l)[i] . | exp(delta[i] . nf(feature_vector(fs,l)))) Note that *sum1* and *sum2* depend on ``delta``; so they need to be re-computed each iteration. The variables ``nfmap``, ``nfarray``, and ``nftranspose`` are used to generate a dense encoding for *nf(ltext)*. This allows ``_deltas`` to calculate *sum1* and *sum2* using matrices, which yields a significant performance improvement. :param train_toks: The set of training tokens. :type train_toks: list(tuple(dict, str)) :param classifier: The current classifier. :type classifier: ClassifierI :param ffreq_empirical: An array containing the empirical frequency for each feature. The *i*\ th element of this array is the empirical frequency for feature *i*. :type ffreq_empirical: sequence of float :param unattested: An array that is 1 for features that are not attested in the training data; and 0 for features that are attested. In other words, ``unattested[i]==0`` iff ``ffreq_empirical[i]==0``. :type unattested: sequence of int :param nfmap: A map that can be used to compress ``nf`` to a dense vector. :type nfmap: dict(int -> int) :param nfarray: An array that can be used to uncompress ``nf`` from a dense vector. :type nfarray: array(float) :param nftranspose: The transpose of ``nfarray`` :type nftranspose: array(float) g-q=i,rRc3*# UH upUv M g7fr#rJrs rrM#calculate_deltas..Ts9.YbS.r%rBr)axisr[) r^onesrr`rr.r$r7r&r`r|outerrS)rrnrmffreq_empiricalrrrrNEWTON_CONVERGE MAX_NEWTONrArr9distr:rrrwrangenumnf_delta exp_nf_deltanf_exp_nf_deltasum1sum2rxn_errors rr~r~sROJ ZZ)3 /F  SZ!23S9A '',__&E%__S8N9.99B)C)R- DIIe$4s$:: * '!ZA*%;;w/({ % 4yy)2yy1,15C INI T)dU22))C$ 678599S[;QQ _ $M#&& Mr!c SnSnSU;aUSnSU;aUSnUc(URSS5n[RXUSS9nOUb [S5e[R "S S 9up[ U S 5n [XXUS 9 SSS5 [R"U 5 /n U /SQ- n U(aU S/- n U(dU S/- n U(a SUS-- nOSnU SSU-S/- n US:aU S/- n SU;a U SSUS-/- n SU;aU SS[US5-/- n [US5(aU S/- n U SU /- n [U 5n[R"U 5 [!XR#5U5nU[$R&"[$R(5-n[+UU5$!,(df  GN-=f![[4an [S U -5U eSn A ff=f![an [S U S!U 35 Sn A NSn A ff=f)"a{ Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the external ``megam`` library. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. :see: ``nltk.classify.megam`` TrrNrr)r$rz$Specify encoding or labels, not bothznltk-prefixw)rrz,Error while creating megam training file: %s)z-nobiasz-repeat10z -explicitz-fvalsr4rBz-lambdaz%.2fz-tunerz-quietrz-maxirHll_deltaz-dppcostz -multilabel multiclasszWarning: unable to delete z: )getrrrtempfilemkstempopenroscloseOSErrorrSrzrremovererrr^rcer)rrrr$rrrrrfdtrainfile_name trainfilerr inv_variancestdoutrs rrrscHIV*%f;' zz.!4 .44 Vt5   ?@@T%--W= .# &) ii '  G ++GK=  H:1144   6L0'::G qyH:VGTF:$6677V FD3vj'9#::;;x  M?" n--G  FB .! "&//*;XFG uzz%''""G Hg ..c' & Z TG!KLRSSTD B *>*:"QC@AABsH"F;7 F)F;G") F83F;;G GG" H,HHc$\rSrSr\S5rSrg)ric URSS5nURSS5nURSS5nURSS5nURSS5nURS S5nURS 5n URS 5n U(d[RXUS 9n[R"S SS9up[R"SS9up[ U S5n[ XU5 UR5 /nURS/5 URSU/5 U(aURSSUS--/5 U (aURSSU -/5 U (aURSS[U 5-/5 URSU /5 URSU/5 US:aURS/5 OURS/5 [U5 [U5n[U5nSSS5 [R"U 5 [R"U5 W[R "[R"5-nU"UU5$!,(df  Nj=f)Nrtao_lmvmrrrr$rrrrrrQznltk-tadm-events-z.gz)rsuffixznltk-tadm-weights-rrz-monitorz-methodz-l2z%.6frBz-max_itz%dz-fatolz -events_inz -params_outz2>&1z-summary)rr/rrrrr rextendrSrrr rrr^rcr)rrrrrrr$sigmarrr trainfile_fdr weightfile_fdweightfile_namerr weightfilers rrTadmMaxentClassifier.trainsJJ{J7  7A&::j$/Hd+ 115zz.!4 ::j)::m,5;;<H(0'7'7&u( $ *2)9)9AU)V& %nc:  i8 |$ 9-.  NNE6E1H#45 6  NNIth7 8  NNHfs8}&<= > n56 78 19 NNF8 $ NNJ< (' / "j(4G# .! /" 5::egg&&8W%%# "s 0 I  I.rJN)rrrrrrrrJr!rrrs5&5&r!rc RSSKnSSKJn U"5n[US35nUR"[ [ URURU5555nSSS5 [US35nURU5nSSS5 [US35nURU5nSSS5 [US35nURU5nSSS5 WWWW4$!,(df  N=f!,(df  Nu=f!,(df  N^=f!,(df  NG=f)Nr) MaxentDecoder /weights.txt /mapping.tab /labels.txt /alwayson.tab) r^ nltk.tabdatarrr{r{mapfloat64txt2list tupkey2dict tab2ivdict) tab_dirr^rmdecr"wgtmpglabaons rload_maxent_paramsrs* ?D  & '1kk$s5==$--2BCDE (  & '1q! (  % &!mmA '  ' (Aooa  ) S#  ( ' ( ' ' & ) (s/?C%5C6DD% C36 D D D&c SSKJn SSKJn SSKJn U"5nU"U5(dU"U5 [ SU35 [US3S5n U RUR[[UR5555 SSS5 [US3S5n U RURU55 SSS5 [US 3S5n U RURU55 SSS5 [US 3S5n U RURU55 SSS5 g!,(df  N=f!,(df  N=f!,(df  Nj=f!,(df  g=f) Nr)mkdir)isdir) MaxentEncoderzSaving Maxent parameters in rrrrr)rros.pathrrrrerwritelist2txtrreprtolist tupdict2tab ivdict2tab) rrrrrrrrmencr"s rsave_maxent_paramsr2s"* ?D >> g ( 23  & , 4==T3::D>!"E"E "E1> E  E E.1 E?czSSKJn SSKJn U"S5n[ U5up4pV[ [ XTUS9U5nU"US9$)Nr)find)ClassifierBasedPOSTaggerz.taggers/maxent_treebank_pos_tagger_tab/english)r)rn) nltk.datarnltk.tag.sequentialrrrr)rrrrrrrmcs rmaxent_pos_taggerrIsE<CDG+G4Cc #CDc B $r 22r!c<SSKJn U"[R5ng)Nr) names_demo)nltk.classify.utilrrr)rrns rdemorXs-,223Jr!__main__)rNN)rNNr)z/tmp)1rr^ ImportErrorrr collectionsrnltk.classify.apirnltk.classify.megamrrrnltk.classify.tadmrr r rr r r rrnltk.probabilityr nltk.utilr __docformat__rrbrrrrr/r@rr]rgrrzr~rrrrrrrrJr!rrs:,Z  #)QQMMFF'/! JA{JA\$4 F$F$R,**@,*^G/"8G/T:D-:Dz5/%@5/pa/!7a/T04_D &04Yx38{PKLT/x7&+7&~,+. 34  zFM1  sC""C+*C+