First, math requires more than just some paper and ink. You need background knowledge, usually in the form of some kind of formal education, to make any kind of advance in the field. It also helps to have access to a library and to other mathematicians of your time.

Second, Fermat had more than just some paper and ink. He had that formal education, access and connections. He was building on the work of others.

Third, up until well into the twentieth century, women lacked the opportunities men had on all three fronts. Most girls received no formal education at all, or it was limited to subjects deemed appropriate for women of the time. (Sophie Germain, for example, had to study math at night in her bedroom because her father had forbidden her from doing so.) They were usually barred from any kind of advanced education in universities, which meant that they also did not have access to the libraries and professors. The few women who did make advances in math usually did so only after long struggles to be educated, or they had other qualities (such as being rich or having a scholar for a father) that helped them to overcome the hurdles their contemporaries faced.

Even today, the playing field is often not level for men and women. To say that social prejudice had nothing to do with the lack of women at the top of mathematics ignores a continuing history of discrimination.