R 中的 NLP BERT 与 tensorflow/Keras 设置

Question

我试图让 BERT 在 R 中运行。

我用 Keras 完成了其他 NLP 任务（例如 word2vec），所以一般设置应该没问题。

我从这里修改了模型代码：https://towardsdatascience.com/hugging-face-transformers-fine-tuning-distilbert-for-binary-classification-tasks-490f1d192379

问题是如何正确插入输入（令牌）。我尝试了很多不同的方法来转换它们（如张量、各种形式的数组等），但似乎无法弄清楚什么样的数据结构/类型/形状应该作为输入。

这是一个简化的、可复制的示例：

#rm(list=ls())
packages <- c("reticulate", "keras", "tensorflow", "tfdatasets", "tidyverse", "data.table")
for (p in packages) if (!(p %in% installed.packages()[,1])) install.packages(p, character.only = TRUE) else require(p, character.only = TRUE)
rm(packages, p)

#reticulate::install_miniconda(force = TRUE) # 1time
reticulate::use_condaenv("~/.local/share/r-miniconda") # win? reticulate::use_condaenv("r-miniconda")

Sys.setenv(TF_KERAS=1) 
tensorflow::tf_version() # install_tensorflow() if NULL
reticulate::py_config()

#reticulate::py_install('transformers', pip = TRUE)
#reticulate::py_install('torch', pip = TRUE)
transformer = reticulate::import('transformers')
tf = reticulate::import('tensorflow')
builtins <- import_builtins() #built in python methods

set.tf.repos <- "distilbert-base-german-cased"

tokenizer <- transformer$AutoTokenizer$from_pretrained(set.tf.repos)  # 
tokenizer_vocab_size <- length(tokenizer$vocab)

###### load model
model_tf = transformer$TFDistilBertModel$from_pretrained(set.tf.repos, from_pt = T, trainable = FALSE)
model_tf$config

# set configs
model_tf$config$output_hidden_states = TRUE
summary(model_tf)

###### data & tokens #####
data <- data.table::fread("https://raw.githubusercontent.com/michael-eble/nlp-dataset-health-german-language/master/nlp-health-data-set-german-language.txt", encoding = "Latin-1")
txt <- data$V1
y <- data$V2
table(y, exclude = NULL)

set.max_length = 100
tokens <- tokenizer(
  txt,
  max_length = set.max_length %>% as.integer(),
  padding = 'max_length', #'longest' #implements dynamic padding
  truncation = TRUE,
  return_attention_mask = TRUE,
  return_token_type_ids = FALSE
)
#tokens[["input_ids"]] %>% str()
#tokens[["attention_mask"]] %>% str()

tokens <- list(tokens[["input_ids"]], tokens[["attention_mask"]])
str(tokens)



####### model ########
input_word_ids <- layer_input(shape = c(set.max_length), dtype = 'int32', name = "input_word_ids")
input_mask <- layer_input(shape = c(set.max_length), dtype = 'int32', name = "input_attention_mask")
#input_segment_ids <- layer_input(shape = c(max_len), dtype = 'int32', name="input_segment_ids")

last_hidden_state <- model_tf(input_word_ids, attention_mask = input_mask)[[1]]
cls_token <- last_hidden_state[, 1,]

output <- cls_token %>%
  layer_dense(units = 32, input_shape = c(set.max_length, 768), activation = 'relu') %>%
  layer_dense(units = 1, activation = 'sigmoid')

model <- keras_model(inputs = list(input_word_ids, input_mask), outputs = output)

model %>% compile(optimizer = "adam",
                  loss = "binary_crossentropy"
)

history = model %>%
  keras::fit(
    x = list(input_word_ids = tokens$input_ids, input_mask = tokens$attention_mask),
    y = y,
    epochs = 2,
    batch_size = 256,
    #metrics = "accuracy",
    validation_split = .2
  )

错误信息：

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: Failed to find data adapter that can handle input: (<class 'dict'> containing {"<class 'str'>"} keys and {"<class 'NoneType'>"} values), <class 'numpy.ndarray'>

Detailed traceback:
  File "/home/sz/.local/share/r-miniconda/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/sz/.local/share/r-miniconda/lib/python3.9/site-packages/keras/engine/data_adapter.py", line 984, in select_data_adapter
    raise ValueError(

提前谢谢了！

Answer 1

您的 model$inputs 形状与您在 fit() 中输入的输入不匹配。

创建一个 TF 数据集很有帮助，因此您可以明确说明您的训练数据集张量形状，并确保这些张量形状与 model$inputs 匹配。

将您的 fit() 调用更改为此使其工作：

x_ds <- tensor_slices_dataset(tokens) 
y_ds <- tensor_slices_dataset(y)

ds <- zip_datasets(x_ds, y_ds) %>% 
  dataset_batch(256)

history = model %>% fit(ds, epochs = 2)