sign language converter using kinect xbox 360 camera

[myra924]

---

Good day! we're on our proj. design, our project is all about converting sign language of the deaf person to text and voice using C# language and kinect....now we're asking for some advice of our project. like recording motions capture by the kinect and assigning the recorded motion with its corresponding name and save it to the database...guys we really need your help and advice on how to solve our problem..Your help will be appreciated and we promise to acknowledge you in our proj.THANKS IN ADVANCE. GOD BLESS!

[cebugdev]

well, well well,
what do we have here? hhehe
what is this? project school thesis?

I do Kinect development and i can help you BUT I'm busy as of the moment.
BUT i'd like to ask questions first.
1. How much is your experience with regards to graphics theory and development specially in C#?
2. How much is your knowledge in Kinect API?
3. How much is your knowledge in C# (OO)
4. How much is your knowledge in vectors and matrix maths ?
5. How much is your knowledge for DB access and management?
6. How much is your knowledge on words to speech algorithms and theories?
7. How much is your experience for implementing synthesizer technologies?

9. Where are you now with this project?

If any answer to the above question is less than a year, then all i can say is "YOu have bitten a project more than you can chew".

I have more than 3 years of kinect development experience and 10 years of graphics development both in applications and games (2D/3D), but even me i cant say that i can implement that kind of project with ease and in one year.

here is my analysis with this project.
1. Detecting sign language is not easy even using kinect, although people already implemented it, but implementing it from scratch is no joke. Each motion has its own algorithm to detect the words it convey.
and imagine there are hundreds of words you need to detect, that means hundreds of algorithm to STUDY and implement.
and needs hundreds to thousands of hours.

Imagine this: in kinect, simple motion such as swiping by left hand, or swiping by both hands needs two different algorithms and if they share similar algo, they still need to handle the part that they differ.

so if this is a school project, one year thesis is Impossible for both research AND implementation.

2. Now let's talk about another ADVANCE skill, synthesizer,
aside from recording ALL the voice necessary you need to sew in those words together (or you can just use free ones if there is) OR if sewing is not an option, you need to re-arrange the words to make more sense.

3. Words combinations, in sign language, sometimes TWO hand gestures means one word or pertains to one meaning. so that means you need to have number 1 and number 2 working.

4. ANother important part is PROCESSING SPeed, IF you do all the processing, computations and conversions in CPU, it will be slow and cannot do runtime conversion. Multi-threading wont help since it still runs on the same CPU albiet different cores.
Get this: dba ang kinect in xbox or in PC sometimes slow to react when it comes to motion? consider that!
Dance games in Xbox do post processing and hit points detection (you need to hit a certain matrix point) to get the score.
Plus Dance games only do motion detect and no further conversion and search in DB. Also Dance games only Detects and COMPARE current motion from user based to what is the current motion on screen.
Also note that Xbox has a separate Hardware for doing Math computations such as those that involves Vectors and Matrices.

Now look at this flow:
(1)user acts a motion (one motion) -> (2)kinect capture motion-> (3) application detects motion capture (4) Application check in the DB on which Capture corresponds to which word, (5) Application detects word (6) application gets the voice data (7) application plays the voice data.

In number 3, you will have hundreds of conditional statements or if using strategy pattern, 100s of call to each motion capture algorithm UNTIL one returns true. At this time, there is a posiblity that the User already moved and acted its second motion.
Same goes for 4.
In 6, dpending on the voice play API, the File handler of the language (or the .NET VM) needs to read the entire voice file before playing the audio(buffering). Games do this by pre-loading the audio on the Load screen. But they only Load those that will be used on specific game level, But how will you know which audio to preload for text to speech?

So if this is a school project, my advice is to DROP it and think of another one.

Why do most students always try to impress people by doing projects they have no idea what to do, its like risking thier chance to graduate. hehee?

Note: if you really want to dive with this project, then be my guest and drop everything you are doing now including eating, going out to parties and all your minor subjects, its gonna be a Hell of a ride.

Drop me a PM from time to time.

Plus if you really want to ask for help, be specific on which part, people will help you if you ask specific questions.
Example: How can I extract motion capture information from kinect? (subject)
content: I want to extract information of both hands only, how do i do this.

And not: How to make robot dance in kinect.
content: I want my robot to dance MAcarena, how?

[manters2000]

well, well well,
what do we have here? hhehe
what is this? project school thesis?

I do Kinect development and i can help you BUT I'm busy as of the moment.
BUT i'd like to ask questions first.
1. How much is your experience with regards to graphics theory and development specially in C#?
2. How much is your knowledge in Kinect API?
3. How much is your knowledge in C# (OO)
4. How much is your knowledge in vectors and matrix maths ?
5. How much is your knowledge for DB access and management?
6. How much is your knowledge on words to speech algorithms and theories?
7. How much is your experience for implementing synthesizer technologies?

9. Where are you now with this project?

If any answer to the above question is less than a year, then all i can say is "YOu have bitten a project more than you can chew".

I have more than 3 years of kinect development experience and 10 years of graphics development both in applications and games (2D/3D), but even me i cant say that i can implement that kind of project with ease and in one year.

here is my analysis with this project.
1. Detecting sign language is not easy even using kinect, although people already implemented it, but implementing it from scratch is no joke. Each motion has its own algorithm to detect the words it convey.
and imagine there are hundreds of words you need to detect, that means hundreds of algorithm to STUDY and implement.
and needs hundreds to thousands of hours.

Imagine this: in kinect, simple motion such as swiping by left hand, or swiping by both hands needs two different algorithms and if they share similar algo, they still need to handle the part that they differ.

so if this is a school project, one year thesis is Impossible for both research AND implementation.

2. Now let go to another ADVANCE skill, synthesizer,
aside from recording ALL the voice necessary you need to sew in those words together (or you can just use free ones if there is) OR if sewing is not an option, you need to re-arrange the words to make more sense.

3. Words combinations, in sign language, sometimes TWO hand gestures means one word or pertains to one meaning. so that means you need to have number 1 and number 2 working.

4. ANother important part is PROCESSING SPeed, IF you do all the processing, computations and conversions in CPU, it will be slow and cannot do runtime conversion. Multi-threading wont help since it still runs on the same CPU albiet different cores.
Get this: dba ang kinect in xbox or in PC sometimes slow to react when it comes to motion? consider that!
Dance games in Xbox do post processing and hit points detection (you need to hit a certain matrix point) to get the score.
Plus Dance games only do motion detect and no further conversion and search in DB. Also Dance games only Detects and COMPARE current motion from user based to what is the current motion on screen.
Also note that Xbox has a separate Hardware for doing Match computations such as those that involves Vectors and Matrices.

Now look at this flow:
(1)user acts a motion (one motion) -> (2)kinect capture motion-> (3) application detects motion capture (4) Application check in the DB on which Capture corresponds to which word, (5) Application detects word (6) application gets the voice data (7) application plays the voice data.

In number 3, you will have hundreds of conditional statements or if using strategy pattern, 100s of call to each motion capture algorithm UNTIL one returns true. At this time, there is a posiblity that the User already moved and acted its second motion.
Same goes for 4.
In 6, dpending on the voice play API, the File handler of the language (or the .NET VM) needs to read the entire voice file before playing the audio(buffering). Games to this by pre-loading the audio on the Load screen. But they only Load those that will be used on specific game level, But how will you know which audio to preload for text to speech?

So if this is a school project, my advice is to DROP it and think of another one.

Why do most students always try to impress people by doing projects they have no idea what to do, its like risking thier chance to graduate. hehee?

Note: if you really want to dive with this project, then be my guest and drop everything you are doing now including eating, going out to parties and all your minor subjects, its gonna be a Hell of a ride.

Drop me a PM from time to time.

Plus if you really want to ask for help, be specific on which part, people will help you if you ask specific questions.
Example: How can I extract motion capture information from kinect? (subject)
content: I want to extract information of both hands only, how do i do this.

And not: How to make robot dance in kinect.
content: I want my robot to dance MAcarena, how?

wow, mighty words from master @cebugdev !