Models
solu-1l
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
1
Layers
2,048
Neurons per Layer
2,048
Total Neurons
3,145,728
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
gelu-1l
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
gelu
Activation Function
1
Layers
2,048
Neurons per Layer
2,048
Total Neurons
3,145,728
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
solu-2l
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
2
Layers
2,048
Neurons per Layer
4,096
Total Neurons
6,291,456
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
gelu-2l
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
gelu
Activation Function
2
Layers
2,048
Neurons per Layer
4,096
Total Neurons
6,291,456
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
solu-3l
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
3
Layers
2,048
Neurons per Layer
6,144
Total Neurons
9,437,184
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
gelu-3l
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
gelu
Activation Function
3
Layers
2,048
Neurons per Layer
6,144
Total Neurons
9,437,184
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
solu-4l
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
4
Layers
2,048
Neurons per Layer
8,192
Total Neurons
12,582,912
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
gelu-4l
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
gelu
Activation Function
4
Layers
2,048
Neurons per Layer
8,192
Total Neurons
12,582,912
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
solu-6l
Model name
neuroscope
Available Services
solu
Activation Function
6
Layers
3,072
Neurons per Layer
18,432
Total Neurons
42,467,328
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
solu-8l
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
8
Layers
4,096
Neurons per Layer
32,768
Total Neurons
100,663,296
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
solu-10l
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
10
Layers
5,120
Neurons per Layer
51,200
Total Neurons
196,608,000
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
solu-12l
Model name
neuroscope
Available Services
solu
Activation Function
12
Layers
6,144
Neurons per Layer
73,728
Total Neurons
339,738,624
Total Parameters
80% C4 (Web Text) and 20% Python Code
Dataset
gpt2-small
Model name
neuron2graph, neuron2graph-search, neuron_explainer, neuroscope
Available Services
gelu
Activation Function
12
Layers
3,072
Neurons per Layer
36,864
Total Neurons
84,934,656
Total Parameters
Open Web Text
Dataset
gpt2-medium
Model name
neuroscope
Available Services
gelu
Activation Function
24
Layers
4,096
Neurons per Layer
98,304
Total Neurons
301,989,888
Total Parameters
Open Web Text
Dataset
gpt2-large
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
gelu
Activation Function
36
Layers
5,120
Neurons per Layer
184,320
Total Neurons
707,788,800
Total Parameters
Open Web Text
Dataset
gpt2-xl
Model name
neuron_explainer, neuroscope
Available Services
gelu
Activation Function
48
Layers
6,400
Neurons per Layer
307,200
Total Neurons
1,474,560,000
Total Parameters
Open Web Text
Dataset
solu-1l-pile
Model name
neuroscope
Available Services
solu
Activation Function
1
Layers
4,096
Neurons per Layer
4,096
Total Neurons
12,582,912
Total Parameters
The Pile
Dataset
solu-4l-pile
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
4
Layers
2,048
Neurons per Layer
8,192
Total Neurons
12,582,912
Total Parameters
The Pile
Dataset
solu-6l-pile
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
6
Layers
3,072
Neurons per Layer
18,432
Total Neurons
42,467,328
Total Parameters
The Pile
Dataset
solu-2l-pile
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
2
Layers
2,944
Neurons per Layer
5,888
Total Neurons
12,812,288
Total Parameters
The Pile
Dataset
solu-8l-pile
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
8
Layers
4,096
Neurons per Layer
32,768
Total Neurons
100,663,296
Total Parameters
The Pile
Dataset
solu-10l-pile
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
solu
Activation Function
10
Layers
5,120
Neurons per Layer
51,200
Total Neurons
196,608,000
Total Parameters
The Pile
Dataset
pythia-70m
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
gelu
Activation Function
6
Layers
2,048
Neurons per Layer
12,288
Total Neurons
18,874,368
Total Parameters
The Pile
Dataset
pythia-160m
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
gelu
Activation Function
12
Layers
3,072
Neurons per Layer
36,864
Total Neurons
84,934,656
Total Parameters
The Pile
Dataset
pythia-350m
Model name
neuron2graph, neuron2graph-search, neuroscope
Available Services
gelu
Activation Function
24
Layers
4,096
Neurons per Layer
98,304
Total Neurons
301,989,888
Total Parameters
The Pile
Dataset
Features
| Feature | Description | Source |
|---|---|---|
| Neuron2Graph | Vizualize the activation patterns of neurons as a graph. Each path through the graph is a n-gram which activates the neuron. From this we also derive a set of similar neurons, which are neurons whose graphs are sufficiently similar. | Paper |
| Neuroscope | Shows how much the neuron activates to each token in a series of text examples. The examples chosen are the examples with the highest activations for that neuron. | Website |
| Neuron explanation | An attempt by GPT-4 to explain what concept the neuron activates on. Only available for
models gpt2-small and
gpt2-xl. | Website |