Hidden Token Annotation

This annotation type allows for a hidden token layer in the document. Hidden tokens are ignored for most intents and purposes but may serve a purpose when annotations on implicit tokens is required, for example as targets for syntactic movement annotation.

Specification

Annotation Category:
 

Structure Annotation

Declaration:

<hiddentoken-annotation set="..."> (note: set is optional for this annotation type; if you declare this annotation type to be setless you can not assign classes)

Version History:
 

Since v2.0

Element:

<hiddenw>

API Class:

Hiddenword (FoLiApy API Reference)

Required Attributes:
 
Optional Attributes:
 
  • xml:id – The ID of the element; this has to be a unique in the entire document or collection of documents (corpus). All identifiers in FoLiA are of the XML NCName datatype, which roughly means it is a unique string that has to start with a letter (not a number or symbol), may contain numbers, but may never contain colons or spaces. FoLiA does not define any naming convention for IDs.
  • set – The set of the element, ideally a URI linking to a set definition (see Set Definitions (Vocabulary)) or otherwise a uniquely identifying string. The set must be referred to also in the Annotation Declarations for this annotation type.
  • class – The class of the annotation, i.e. the annotation tag in the vocabulary defined by set.
  • processor – This refers to the ID of a processor in the Provenance Data. The processor in turn defines exactly who or what was the annotator of the annotation.
  • annotator – This is an older alternative to the processor attribute, without support for full provenance. The annotator attribute simply refers to the name o ID of the system or human annotator that made the annotation.
  • annotatortype – This is an older alternative to the processor attribute, without support for full provenance. It is used together with annotator and specific the type of the annotator, either manual for human annotators or auto for automated systems.
  • confidence – A floating point value between zero and one; expresses the confidence the annotator places in his annotation.
  • datetime – The date and time when this annotation was recorded, the format is YYYY-MM-DDThh:mm:ss (note the literal T in the middle to separate date from time), as per the XSD Datetime data type.
  • n – A number in a sequence, corresponding to a number in the original document, for example chapter numbers, section numbers, list item numbers. This this not have to be an actual number but other sequence identifiers are also possible (think alphanumeric characters or roman numerals).
  • textclass – Refers to the text class this annotation is based on. This is an advanced attribute, if not specified, it defaults to current. See Text class attribute (advanced).
  • space – This attribute indicates whether spacing should be inserted after this element (it’s default value is always yes, so it does not need to be specified in that case), but if tokens or other structural elements are glued together then the value should be set to no. This allows for reconstruction of the detokenised original text.
  • src – Points to a file or full URL of a sound or video file. This attribute is inheritable.
  • begintime – A timestamp in HH:MM:SS.MMM format, indicating the begin time of the speech. If a sound clip is specified (src); the timestamp refers to a location in the soundclip.
  • endtime – A timestamp in HH:MM:SS.MMM format, indicating the end time of the speech. If a sound clip is specified (src); the timestamp refers to a location in the soundclip.
  • speaker – A string identifying the speaker. This attribute is inheritable. Multiple speakers are not allowed, simply do not specify a speaker on a certain level if you are unable to link the speech to a specific (single) speaker.
Accepted Data:

<alt> (Alternative Annotation), <altlayers> (Alternative Annotation), <comment> (Comment Annotation), <correction> (Correction Annotation), <desc> (Description Annotation), <external> (External Annotation), <metric> (Metric Annotation), <part> (Part Annotation), <ph> (Phonetic Annotation/Content), <ref> (Reference Annotation), <relation> (Relation Annotation), <str> (String Annotation), <t> (Text Annotation)

Valid Context:

<def> (Definition Annotation), <event> (Event Annotation), <ex> (Example Annotation), <head> (Head Annotation), <note> (Note Annotation), <p> (Paragraph Annotation), <quote> (Quote Annotation), <ref> (Reference Annotation), <s> (Sentence Annotation), <term> (Term Annotation), <utt> (Utterance Annotation)

Explanation

Hidden tokens are tokens that are explicitly not part of the original text. They are either implied or tokens that act as a dummy for further linguistic annotation. Hidden tokens are are valid target for any form of span annotation through the <wref> element (see :ref:`span_annotation_category`_). They are structure elements so may appear interleaved with the normal tokenisation layer, for which the order is significant.

Example

The following example shows syntactic movement annotation which makes use of a hidden token for an implicit subject.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
<?xml version="1.0" encoding="utf-8"?>
<FoLiA xmlns="http://ilk.uvt.nl/folia" version="2.0" xml:id="example">
  <metadata>
      <annotations>
          <token-annotation set="https://raw.githubusercontent.com/LanguageMachines/uctodata/master/setdefinitions/tokconfig-eng.foliaset.ttl">
			 <annotator processor="p1" />
		  </token-annotation>
          <hiddentoken-annotation>
			 <annotator processor="p1" />
		  </hiddentoken-annotation>
          <text-annotation>
			 <annotator processor="p1" />
          </text-annotation>
          <sentence-annotation>
			 <annotator processor="p1" />
          </sentence-annotation>
          <pos-annotation set="adhoc">
			 <annotator processor="p1" />
          </pos-annotation>
          <syntax-annotation set="adhoc">
			 <annotator processor="p1" />
          </syntax-annotation>
          <description-annotation>
			 <annotator processor="p1" />
          </description-annotation>
      </annotations>
      <provenance>
         <processor xml:id="p1" name="proycon" type="manual" />
      </provenance>
  </metadata>
  <text xml:id="example.text">
    <s xml:id="example.s.1">
      <t>Isn't a whole lot left.</t>
        <hiddenw xml:id="example.s.1.w.0">
            <t>*exp*</t>
            <desc>empty expletive subject</desc>
            <pos class="EX" />
        </hiddenw>
        <w xml:id="example.s.1.w.1" space="no">
            <t>Is</t>
            <pos class="BEP" />
        </w>
        <w xml:id="example.s.1.w.2">
            <t>n't</t>
            <pos class="NEG" />
        </w>
        <w xml:id="example.s.1.w.3">
            <t>a</t>
            <pos class="D" />
        </w>
        <w xml:id="example.s.1.w.4">
            <t>whole</t>
            <pos class="ADJ" />
        </w>
        <w xml:id="example.s.1.w.5">
            <t>lot</t>
            <pos class="N" />
        </w>
        <w xml:id="example.s.1.w.6" space="no">
            <t>left</t>
            <pos class="VAN" />
        </w>
        <w xml:id="example.s.1.w.7">
            <t>.</t>
            <pos class="PUNC" />
        </w>
        <syntax>
            <su xml:id="example.s.1.su.1" class="IP-MAT">
                <su xml:id="example.s.1.su.2" class="NP-SBJ">
                    <wref id="example.s.1.w.0" />
                </su>
                <su xml:id="example.s.1.su.3" class="VP">
                    <su xml:id="example.s.1.su.4" class="BEP">
                        <wref id="example.s.1.w.1" />
                    </su>
                    <su xml:id="example.s.1.su.5" class="NEG">
                        <wref id="example.s.1.w.2" />
                    </su>
                    <su xml:id="example.s.1.su.6" class="VP">
                        <su xml:id="example.s.1.su.7" class="NP-LGS">
                            <wref id="example.s.1.w.3" />
                            <su xml:id="example.s.1.su.8" class="ADJP">
                                <wref id="example.s.1.w.4" />
                            </su>
                            <wref id="example.s.1.w.5" />
                        </su>
                        <wref id="example.s.1.w.6" />
                    </su>
                </su>
                <su class="PUNC">
                    <wref id="example.s.1.w.7" />
                </su>
            </su>
        </syntax>
    </s>
  </text>
</FoLiA>