add adaptershare

Author: Bei Chen

README.md

@@ -10,6 +10,7 @@ This repo hosts multiple open-source codes of the [Microsoft Research Asia DKI G
 - October, 2022: our paper [Towards Knowledge-Intensive Text-to-SQL Semantic Parsing with Formulaic Knowledge](https://arxiv.org/abs/2301.01067) was accepted by EMNLP 2022.
 - October, 2022: our paper [Reasoning Like Program Executors](https://arxiv.org/abs/2201.11473) was accepted by EMNLP 2022.
 - October, 2022: our paper [LEMON: Language-Based Environment Manipulation via Execution-guided Pre-training](https://arxiv.org/abs/2201.08081) was accepted by EMNLP 2022 Findings.
+- October, 2022: our paper [AdapterShare: Task Correlation Modeling with Adapter Differentiation](https://aclanthology.org/2022.emnlp-main.728.pdf) was accepted by EMNLP 2022.
 - Septempter, 2022: our paper [LogiGAN: Learning Logical Reasoning via Adversarial Pre-training](https://arxiv.org/abs/2205.08794) was accepted by NeurIPS 2022.
 - March, 2022: Our paper [Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation](https://aclanthology.org/2022.acl-long.142.pdf) was accepted by ACL 2022.
 - August, 2021: Our paper [Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing](https://aclanthology.org/2021.findings-acl.100.pdf) was accepted by ACL 2021 Findings.

adaptershare/README.md

@@ -0,0 +1,37 @@
+<h1 align="center"> AdapterShare: Task Correlation Modeling with Adapter Differentiation </h1>
+
+<div align=center><img width="350" height=350" src="assets/adaptershare.png"/></div>
+
+## Introduction
+Thanks to the development of pre-trained language models, multitask learning (MTL) methods have achieved great success in natural language understanding. However, current MTL methods pay more attention to task selection or model design to fuse as much knowledge as possible, while the intrinsic task correlation is often neglected. It is important to learn sharing strategies among multiple tasks rather than sharing everything. In this paper, we propose AdapterShare, an adapter differentiation method to explicitly model task correlation among multiple tasks. AdapterShare is automatically learned based on the gradients on tiny held-out validation data. Compared to single-task learning and fully shared MTL methods, our proposed method obtains obvious performance improvements. Compared to the existing MTL method AdapterFusion, AdapterShare achieves an absolute average improvement of 1.90 points on five dialogue understanding tasks and 2.33 points on NLU tasks.
+
+## Quickstart
+
+### Setup Environment
+#### Install via pip:
+1. install requirements </br>
+   ```> pip install -r requirements.txt```
+
+2. install our modified `adapters` package </br>
+   ```> pip install git+https://github.com/WowCZ/adapter-transformers.git```
+
+
+#### prepare the nlu dataset:
+1. Download data </br>
+   ```> sh download.sh``` </br>
+   Please refer to download GLUE dataset: https://gluebenchmark.com/
+
+2. Preprocess data </br>
+   ```> sh experiments/glue/prepro.sh```
+
+
+### Training scripts:
+1. Individual training script (baseline multitask learning): </br>
+    ```> sh scripts/adapter_train.sh -tr wnli -te wnli -ls 50 -ss 100  > LOGS/mtnlu_wnli.log``` </br>
+
+2. AdapterShare training script: </br>
+    ```> sh scripts/adapter_train.sh -tr mnli,cola,qnli,qqp,rte,sst,stsb,wnli -te mnli_matched,mnli_mismatched,cola,qnli,qqp,rte,sst,stsb,wnli -ls 1000 -ss 2000  > LOGS/mtnlu_unified.log``` </br>
+
+
+## Claim
+In this repository, the code of NLU dataset collection and baseline multitask learning is based on https://github.com/namisan/mt-dnn. The AdapterShare training process is modified from https://github.com/adapter-hub/adapter-transformers.

adaptershare/_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-minimal