Okay, I'm going to take a stab at this.
mythology vs. folklore
To me, folklore is more localized to one geographic area or span of time, whereas mythology tends to be more encompassing. In addition, mythologies tend to have an overall structure, whereas folklore tends to be mostly composed of independent stories. I don't see a particularly good reason to exclude folklore from the scope of this site, however, because their natures are rather similar. That is, they have a similar "feel", so to speak. The "feel" of being fantastical. So, it's like a particular culture's folklore is a vague mythology that is...less grandiose and overarching than the "traditional" mythologies we think of.
For this, I think it would be useful and good to have a rule of thumb based on notability or similar concept. Skeptics.SE requires that a claim be notable - that is, there is proof the claim was made and that it wasn't just some random guy on the internet. Christianity.SE has something somewhat similar in that questions about denominations or beliefs with little or no external evidence are discouraged. I would say that folklore should be on-topic and in-scope, but that producing a good question should be a little harder and take more work. Questions need to be interesting and at least somewhat objectively answerable.
This aspect of this site's scope is definitely something I can see evolving over time. Also, very relevant Wikipedia page.
Examples:
- Did the events in Hansel and Gretel actually happen?
Off-topic! Avoid questions of validity!
- Why did children believe the story of Hansel and Gretel, and why did adults continue telling it?
Off-topic. This is really more of a sociology/psychology question.
- Was Hansel's strategy of putting down a trail of breadcrumbs inspired by the Greek myth of the Minotaur?
On-topic...maybe. Might not be a good question, but I'd consider it acceptable.
- What is the likely origin of the fairy tale Hansel and Gretel?
On-topic. In part because there isn't another site for folklore, and I doubt a separate site would ever be created that could handle questions like these. Except maybe History.SE.
mythology vs. religion
Okay, I am a moderator on Christianity.SE and I believe that most events spoken of in the Bible actually happened, yet I still believe that Christianity has a mythology. C.S. Lewis calls Christianity a true myth, so I do not believe that mythology has to be considered false, inaccurate, or incorrect. Hence, in my view, all religions have an associated mythology.
My proposal in this regard is that if a question is specifically about a mythological aspect of a religion, then it is in scope for this site. A question about Creationism (especially Young Earth Creationism) would be out of scope for this site, unless it deals specifically with the origin myth aspect of it. I'm not sure how such a question would be worded though.
Also in scope would be comparing, say, the resurrection myth of Jesus Christ with the resurrection myth of Horus. I personally believe the former is true and the latter is false, but I'm still perfectly fine with calling both of them "myths". I think a useful rule of thumb might be if it's in-scope at <Religion>.SE, it's off-topic here.
We should definitely avoid questions of validity. Avoid them like the plague!
Examples:
- Jesus never existed. He's just a myth. Why do people believe he did?!?
Off-topic. If I were a mod, I'd close immediately and probably delete not long after. (Questions like Was Jesus a myth? should similarly be burninated.)
- Why were the apostles so enthusiastic about spreading the myth of Jesus Christ?
Off-topic. Would fit better on Christianity.SE, and this is not really directly about mythology. Same goes for similar questions for other religions.
- Was the myth of Jesus Christ's resurrection inspired by the myth of <insert god here>'s resurrection?
On-topic. May somewhat overlap with Skeptics.SE, so answerers on this site should also pay attention to mythological aspects, rather than just focusing on historical juxtaposition and the like.
- How much did the myth of Gilgamesh, especially the global flood, inspire the Flood myths of other cultures?
On-topic. Might be too primarily-opinion-based, so might be closed for that reason.
mythology vs. fiction
One big distinction between these is that fiction is considered by few to be true, whereas mythologies typically have adherents that whole-heartedly believe in the stories. If someone's belief system is at least partially not science- or fact-based and they are genuinely and sincerely willing to die for it, there's a pretty good chance that belief system is a religion and/or a mythology. There might be a few such people who would genuinely stake their life on Star Wars being true and factual - of which I am not one (Lord help my unbelief) - but they are few in number and generally not considered to be sane and intelligent. A minor point: there is definitely a Star Wars mythology (feel free to swap in your favorite fictional universe), but it is universally (ha) considered only in an in-universe context, so questions dealing with the Star Wars mythos would go on SciFi.SE and similar StackExchange sites.
In contrast, people lived and died in the context of great mythologies such as the Greeks, Egyptians, and Norsemen, to name a few big examples. This is also true of Judaism, Christianity, Islam, Buddhism, Hinduism, and all sorts of other religions, which adds support to my stance that the mythological aspects of religions should be on-topic for this site.
Meta-questions about fictional mythologies, however, might be on-topic if phrased appropriately. Questions asking about some mythological characteristic of some fictional story, for instance.
Examples:
- Who were the first people to use the Force in Star Wars?
Off-topic. This would go on SciFi.SE, not here. This is an in-universe question.
- What are the major components of the mythology of the Na'vi in Avatar?
Off-topic. Though in a funny way. I would say that this should go on SciFi.SE, but if enough people in this community decide otherwise, then so be it.
- Is the Hero's Journey a common device in myths?
On-topic. I was thinking specifically of Luke Skywalker in Star Wars, but you could ask much the same about many other fictional characters. A good answer would probably bring up Joseph Campbell's work in this area.
- Why, according to Joseph Campbell, is the monomyth is so common?
On-topic. Well, I'm a bit biased since I just asked that question, but I do think it's a good on-topic question for this site.
- How did George Lucas incorporate mythological elements into Star Wars?
On-topic. At least, I'd say so. Lucas has given interviews where he discussed the mythological aspects of his story, and this is out-of-universe.
other characteristics
Should Loch Ness, Bigfoot, and UFOs be considered myths? I would say no. Myths tend to be stories with characters, as far as I can tell. These examples are simply fantastical creatures/entities that are believed to exist by some. Not much more than that. An okay question might be one like: Does <myth> mention what we would now call UFOs? I think it'd be hard to successfully pose a question like that. On top of that, such things tend to be more current, whereas mythologies tend to be more historical/ancient.
Examples:
- Where is Bigfoot considered to be hiding?
Off-topic. I dunno where this question would go, but I wouldn't leave it open on this site.
- Does <myth> mention what we would now call UFOs?
On-topic. Maybe. Depends on how well the question is written.
Addendum: Luna contributed a very good comment:
Re: the last distinction about Nessie, UFOs etc. - it may be worth stating that while folklore is on-topic, cryptozoology is not. So, questions about the legend, story or mythos of Bigfoot might be relevant (I believe there are stories about this creature and similar ones that predate the sightings/discovery obsession), but questions about his existence or sightings would certainly not be.
Folklore = on-topic. Cryptozoology = off-topic.
As always, please discuss this and your opinions in the comments.