Overview
Welcome to harrypotterGPT! This projects implements a mini GPT model
that can write infinite Harry Potter. It's a project focused on
natural language processing and generative AI. The models are trained
on a coherent dataset containing all of the
Harry Potter books. This allows the model to output more Harry Potter inspired text.
Although most language models are implemented as autoregressive models
on the word, or subwords, level, I decided to implement this on a
character based level for the sake of simplicity. Thus, the models
generates one character at a time in an autoregressive fashion.
As a baseline model, I implemented the simplest of language models,
the Bigram model. It makes predictions by only looking at the most
recent character. It is extremely bad at generating sensible text
since it has no knowledge of the characters beyond the preceeding
letter. The output of the Bigram model is below:
Foasthaprse tize herst el O u fZEie hy: Hak, CORineg aggell thrr
Masearor charnge? Tyoucre thy, chouspo in mppry way avend oubur'er
sickes bokecard dhiceny He tw el fe oupise he, lbustselownthous; I m w
T: The at; I m hofaruk mondrn itheland's oe, oghithet f,
badogienthofBRI'sey &CleDWeer'dsureisold array n ICoyockind m murs, in
mamybalorenyongmyooe, d Vofetthindy st Hefqu brveseay alsteanerm to,
oupomp rede d pre h, gavitYOfrrerean apsts lathind my d erouerse
IOLUED d ngKE hicerire. II IS: I
As the main model, I implemented a decoder transformer model based on
the famous research paper
"Attention is All You Need"
from 2017. The model has the same architecture as the decoder block
presented in the paper. The model has just over 10.7M parameters and
is trained on roughly 6 million tokens. For comparison, GPT-3 has 175
billion parameters and was trained on 300 billion tokens. By simply
looking at the output below, it might seem reasonable. However, when
you start reading it you'll see that it doesn't make much sense. I
believe this is a consequence of the simplicity of the model and the
scarce dataset. I would be able to train a more complex network if I
had more computing power.
among Ron even Galleons robes out standing emblisty fire called Potter
!Harry remoted .Screamed Our naver beard !Between people and Hermione
could all providing and tried to deal Umbridges right imaginant but
the greenhouses breeze rush in the foreign in the mater the useful
creeats perched baths of dangerous common room on the fork his eyes
drovely he flated them a little dull for his own forehead box .Im
sorry said Harry bravely .She thats just weach dear .She must enter
exclasm lessons freement or libyrange need to Scabbers now by our
common room that is ?By the Dark Urders bodyched who hadnt continued
into accomprehenese her barely sure just because she approached the
windows creaking her waistBlack disrovements when we gown and triveled
for two butliling evening except and twenty fortural weekendures
woldaO .All between .To the open .Remember .come here no not said
Sirius .He had hootherly had stopped across the Bludger had been to
give this Order Harry said deeply and door open bag a desk .You havent
speak Harrys might mont Munduffgus goal .Theres just least their faul
we else show Petunia cross and as there said Harry .She stands still
shaking all over Ron and Hermione s hand perched Black servant of
satisomacks to face curious if Harry too test you said Ginny angrier
at Harry .Harry asked .Willing you when he read me Hermione looked as
though I locke unfortunately of all the empty latter I am .Right said
Dumbledore after an houseelf rather have somilated you fan you forgot
two .He gave sit full of banks and you can understand enough yourself
at it was Neville .See very harderouses much likea Bill and Galleons
big times to kill for this elf hes rup through both him and whispered
those movement one as the spell mets off to my very worst fun what
sounded inventing we 1 have been Probably Plank so good for you left
?An then Protected seaming spectioner of Dumbledore was never both
since of ten .Fround George cant get Goyle .And youre a genialo as of
the Ministry people of Dobby family hiding go .Potter sir dont worry
!Harry screamed the werewolfs promphle?Viikdouble Dumbledore once
Professor Sprout said Look anxiously .Harry repeated .Their brooms
Harry felt that foot but always the arcivals at the staff for Harry
food the laugh .Were not coming wizard just put looking at Hagrids
bone with laughing wood like !And only Mr Weasley when persuaded her
clear ways too person careless lay flowled forward .And the first
doors had flown three huge with when roasts followens were closed
closed into bowel none of them carried from the ground seated sperring
.Harry come into Slughorns hoom afternoon yet says said Hermione
powerful above Malfoy shostly .Harry has planned to hear a great
student at a lesson at valenting Rons blood looked up theyd must well
attempt to tell them he had been an uggested in Malfoy had getten
concerned himself .You wish it .Malfoy snoog a quarter and I know
Lupin picked out a little danger from cardled up for the moment he
took there but when he would be becareful if hed known he had used the
prophecy and he slung down open his mind .The Witchs alleyhaired
labeled .Yeah blue or he was merely lifted from ninece he was rae .Of
this owl must .he had steal sunk and avotestrolled the gentulies was
stand out the bell of the door .Harry thought What made it up the
little champion down all the most names of the whispers from their
hand a chuckle and his bounches cackled face and Dumbledore tor where
their screemed a potion carriages were birtting .Other words .that was
so very good idea how much up the leaks interest but it was locked in
his head the scream unlikely other man warmuged the females who turned
poularter make which one tiny giant figure .Thousan he was not
expecting a Grimmaul something boy after that noticing with
Brykhardfamily and Ginny went up awake .As price as you are ustend to
me Professor McGonagall go too .Professories protecting ?I doubt a
door than Aragonsleys either theyve said .An inditional the Grimmauld
Place they were building out of the portrait of business tea all Bills
and wimmes he had fallen fainted silenting .It was an inch time alarm
.It was still more a keep ‘He glanced from his very fruer than We can
use just show that Harry Potter unusually sensationally on hiss .Ted
gin nearer interest Apparate with Professor Umbridge ?Harry asked
broadly and your frames .Its not very small iste during magny .Harry
could think that morning was than they were having and having slipped
black greaned into the rohes windows .Sanne !hickled Harry .Now Im
sure .a I think that swork him .Why will Mr Olliver sproute I hiding
you think wed better be near said Ginny thoroshing pinks not .He was
making thinking of greenhouse and vanished by the time there were
looking of blaze and shuff once a rin arm and blew heard it say
demertors innocent thirteen than the one hand he gave a smile .Sirius
said as muttering breakfast as Riddle and Ron bitter friendyly .Yeah
yehll be aile said Hagrid half made himssed it us a cat bat her pup
.Harry let out a quintment and Moody said Tonks shaking handkerch
there .Theyre night theyll not get linger to do with their hours song
eeth out of the green light ayoulelged but someowake did need to take
them .Harry had restered that to the feast after him not said Luping
showering sturface Ron .You knew anyone it was reading to hatrasp that
that said almost full airply to Harry a meeting in the champion with
them .Ebbited !said Hermione slowly .WHarry called a parchment into
the highpitched pairs leaving giant wing steal very more keen on
merrily becoming fast .For something Imppeter Curse insplictement
batlacks an head Ravenclaw threw his hown dark pain Harry was soler
.Harry knew whater the dementors he was covered out the school by
three hundred this time took a dark frozen on the fountain .Hullo with
one shallowed from him up the steps Harry could not toll him .Harry
saw that Fange had gotnicked he run down on the floor between the dent
ferm the staff are twenty feet his brain were a hystle featherous
belone Voldemort .He did not want to look at the wizard before said
Missa !Hogwarts and the sectarious .But theres looking !Harry
transportials faces now let out Ron at the shorarian and gaveling
himselves cat dry lemon emboring .Charms open Harrys clipboards like
Lets Argus Dungs last Friday Took of serveng sweat sticking on his
knife and for Harrry spun for the right wall .Maybe for stream for you
this black effore .His an angry conversation evening wizard bit than
his taes and hears that turned suspiciously over magic however when
everyone while there !May Doges a sight nose in the deal to stay up
here !Harry could think down that still silk and Crabbe and Goyle
around them to well back and sleepily bewilder than it happened .Mr
Weasley by Mr Weasley at the Durmstrang Cornelie and Snape fell
slightly over her he straightened to the castle giving the bless
without practiven and seemed to make Harry feel at fear Harry couldnt
hell of movement and once at all .One wizards first please in our
Death Eater whose nonstructed perfect but And unwikstan particularly
followed Dumbledores snock that they were hoaded or examinin
Dumbledores kept so low sharps with tables forming very time .The
Dreath Educted .He however began he attempted to motion anybody net at
them but Harry and Ron opened the steps windows to check when the
Honeys light had gone ‘He tries he finished as promission to find the
secret hund him in the heavily of crease his voice fere giving it they
instructed how could Sirius said this cannon .At once the first time
to admance Park and threw him !said Mrs Weasley madly .I dont mink me
Id find that !said Harry looking around Floud .Hissing face ?he said
lunch .She took a door from metalking in the direction of what lit was
growing of silent pear victor .12 really be in there Gryffeor will
find out said Harry a lookout first clenches that had it happened .A
golden fireliesnt like she stamped up a wideaged tight that which
could not be in date that might have been worried by at some feet
.Good the load is carrying this gold place .He does hour .Neither be
and new kind of his fatture into people ?Being his proof toment .We
might have got researched her .Sirius said Dumbledore looking other
.Ill be hopeful have a changed dragon of free .Wed what we know
Professor Dumbledore littered trembling Im heard any practical
teachers tonight this ?Riddle made the jet of the fact that he could
no dear not to know whether that he was still disturbed But Scrimgeour
willn not to distract him .Protective that custartains have been
suddenly impised the dementors with his warm .Harry did not
approaching of the dementors and dare you like to put go in any chance
are gaiting startledoned pointing of Lavender Delacour .7 I with Ron
forced them once Harry !he craned to the noise the right .Gige Must go
to it said Harry and smalling to Hedwigs making noise a something
thing Gryffindor spreaded him every finished and Dumbledore too
Dumbledores office spast from whom happening to Harry who could take
clump off .Yeah did .Sirius treasured Harry and Redita oncertovity as
though I have given in Both for the time Id better groan .What it
worth team though he spots of telling them .He was useful to discuss
you to get rid of my hostbuilt that he did so you have used to be
Perhaps when that matters seemed to be deeply .Invisibility where the
truth that was around Like Fellure .At this at the strain Harry
.Hermione got up felt over the house individently he had never been
making it back and always was happened levely over her and her senset
.Reseived in stee vir in a prophecy !he shouted across the currently
.Injusticed her osquinet in caught .Er at the Gryffindor bottle was
trolles on the sacketblank !Where ?she said smiling .She made the girl
pale .Someones have got away him about the vaste ?Harry greated tanger
.Im taking Imporwarte .If Professor Draco did are my not actiding
.Come on