Listed in Table 1. We will describe these evaluation indicators in detail.Appl. Sci. 2021, 11,7 ofFigure 5. BiLSTM framework. Table 1. Specifics of evaluation metrics. “Auto” and “Human” represent automatic and human evaluations respectively. “Higher” and “Lower” imply the higher/lower the metric, the superior a model performs. Metrics Composite score Good results Price Word Freqency Grammaticality Fluency Naturality Evaluation Technique Auto Auto Auto Auto (Error Rate) Auto (Perplexity) Human (Naturality Score) Improved Greater Greater Larger Reduce Reduced Larger(1) The attack accomplishment rate is defined because the percentage of samples incorrectly predicted by the target model to the total number of samples. Within this experiment, these samples are all connected for the universal trigger. The formula is defined as follows S= 1 Ni =( f (t, xi ) = yi ),N(6)exactly where N is the total quantity of samples, f represents the target model, t represents the universal trigger, xi represents the ith test sample, and yi represents the actual label of xi . (2) We divide it into four components for the top quality of triggers, which includes word frequency [29], grammaticality, fluency, and naturality [23]. The typical frequency of your words within the trigger is calculated using empirical estimates in the education set of the target classifier.Appl. Sci. 2021, 11,eight ofThe higher the average frequency of a word, the additional instances the word seems in the instruction set. Grammaticality is measured by adding triggers in the same number of words to benign text, after which applying an online grammar check tool (Grammarly) to obtain the grammatical error price with the sentence. With all the support of GPT-2 [14], we use Language Model Perplexity (PPL) to measure fluency. Naturalness reflects regardless of whether an adversarial instance is organic and indistinguishable from human-written text. (3) We construct a composite score Q to comprehensively measure the efficiency of our attack strategy. The formula is defined as follows Q = + W – – (7)exactly where S could be the attack accomplishment price of the trigger, W would be the average word frequency of the trigger, G could be the grammatical error price of your trigger, and P could be the perplexity on the GPT-2 [14]. W, G, P are all normalized. , , is definitely the coefficient of each parameter, and + + + = 1. So as to balance the weight of each and every parameter, we set , and to 0.25. The greater the Q score, the greater the attack performance. To Dodecyl gallate Formula further verify that our attack is extra all-natural than the baseline, we carried out a human evaluation study. We supply 50 pairs of comparative texts. Every single group includes a single trigger and a single baseline trigger (with or without the need of benign text). Workers are asked to pick out a much more natural one, and humans are allowed to select an uncertain alternative. For every single instance, we collected 5 various human judgments and calculated the average score. 4.four. Attack Final results Table 2 shows the results of our attack and baseline [28]. We observe that our attack achieves the highest composite score Q on all the two datasets, proving the superiority of our model over baselines. For each optimistic and unfavorable situations, our process has a larger attack success price. It may be located that the achievement price of triggers on SST-2 or IMDB information has reached greater than 50 . Additionally, our method accomplished the most effective attack impact around the Bi-LSTM model Isophorone Cancer trained on the SST-2 information set, with a success price of 80.1 . Comparing the models trained on the two data sets, the conclusion can be drawn: The Bi-LSTM model educated around the SST-2 data set.