You are probably under the impression (which is encouraged by twisted xian 'translations') that the people who were responsible for the calf atrocity were from
Yisroʾel, but actually that is not the case. First of all, look at the chant of the worshippers: "Hey,
Yisroʾel, these are YOUR gods that brought YOU out of Egypt" (
Sh'moth 32:4, 32:8); and then look carefully at what Hashem said to
Moshah: "...YOUR people that YOU brought from Egypt have become corrupt..." (
Sh'moth 32:7,
D'vorim 9:12). Nowhere does
Hashem refer to
Yisroʾel as "
Moshah's people that
Moshah brought from Egypt"—He always calls us "My people that I brought from Egypt". Nevertheless, when the FEW Hebrews who had joined in with the unclean idolaters (the עֵרֶב רַב
ʿerav rav or "great rabble" that
Moshah allowed to escape from Egypt with
Yisroʾel, see
Sh'moth 12:38) had to be punished, the
Torah records in
Sh'moth 32:28 that the number of those killed that day amounted to "about three thousand men" (out of a total adult male population of more than 600,000—i.e., less than half of 1%).
The
Torah says (
Sh'moth 32:4) that it was
ʾAharon who made it, but when he was later telling his brother
Moshah what had happened, he said:
The total population (males AND females) who were present is immaterial because the 3,000 who were killed were only the men:
We also know that in the first census (which, according to
B'midbor 1:1
et seq., was held just eleven months after the calf incident) there were found to be 603,550 adult men (this figure is stated on three separate occasions—
Sh'moth 38:26,
B'midbor 1:46 and
B'midbor 2:32), so that the men who were killed for taking part in the calf-idolatry were 3,000 out of 603,550 =
0.497% of the total adult male population. The 'people' who were 'struck with a plague' were the 'mixed rabble' of slaves—the ones primarily responsible for the outrage.
Take a look at
Sh'moth 32:7-8
The text reads אֵלֶּה אֱלֹהֶיךָ
ʾellah ʾalohacho ('THESE ARE your gods') and even the verb that follows, הֶעֱלוּךָ
haʿalucho, is plural (הֶעֱלוּךָ
haʿalucho is the 3rd person plural, past tense
hifʿil paradigm הֶעֱלוּ
haʿalu 'they brought up' with the 2nd person object-case suffix ךָ־ 'you' added to it); from which the reason for the plural forms is now obvious (the 'mixed rabble' of gentile slaves that
Moshah, in a humanitarian gesture, had allowed to leave Egypt together with the escaping Hebrews, had no understanding of a single G-d and so they would naturally have spoken about 'gods' in the plural). Indeed, when spoken by gentiles, אֱלֹהִים
ʾalohim is occasionally used in the
Tanach in the plural sense of 'the gods' but still referring to G-d Himself: there is an example of this in
Sh'muʾel ʾAlaf 4:6-8 vis-à-vis the
P'lishtim.
Seems like I’m the only one who can bring both viewpoints together 
