Python for Web Scraping in Flask, Jinja 2, SQLite with project setup along with the deployment

on

 Python for Web Scraping in Flask, Jinja 2, SQLite 

Flask (a lightweight web framework), Jinja2 for templates, and SQLite as the database for storing blog posts.

Here's a step-by-step guide:

Step 1: Set Up Your Project

Create a project folder and install the required dependencies.

mkdir python_web_scraping_blog
cd python_web_scraping_blog
python3 -m venv venv
source venv/bin/activate  # for Linux/macOS
# For Windows:
# venv\Scripts\activate
pip install flask flask_sqlalchemy

Step 2: Project Structure

Create the following folder structure:

python_web_scraping_blog/
├── app.py              # Main Flask app file
├── templates/
│   ├── base.html       # Base template
│   ├── index.html      # Homepage
│   └── post.html       # Individual blog post
├── static/
│   ├── css/
│   │   └── style.css   # Stylesheet
└── models.py           # Database models

Step 3: Set Up Flask App (app.py)

from flask import Flask, render_template
from models import db, BlogPost

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///blog.db'
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db.init_app(app)

@app.route('/')
def index():
    posts = BlogPost.query.all()  # Fetch all posts from the database
    return render_template('index.html', posts=posts)

@app.route('/post/<int:post_id>')
def post(post_id):
    post = BlogPost.query.get_or_404(post_id)  # Fetch individual post
    return render_template('post.html', post=post)

if __name__ == '__main__':
    app.run(debug=True)

Step 4: Create Database Models (models.py)

from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy()

class BlogPost(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(200), nullable=False)
    content = db.Column(db.Text, nullable=False)
    created_at = db.Column(db.DateTime, default=db.func.now())

    def __repr__(self):
        return f"<BlogPost {self.title}>"

Step 5: Initialize Database

Before running the app, initialize the database:

from app import app
from models import db

with app.app_context():
    db.create_all()

This creates a blog.db SQLite file with a BlogPost table.

Step 6: Create HTML Templates

base.html (Shared Layout)

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Python Web Scraping Blog</title>
    <link rel="stylesheet" href="{{ url_for('static', filename='css/style.css') }}">
</head>
<body>
    <header>
        <h1>Python Web Scraping Blog</h1>
        <nav>
            <ul>
                <li><a href="/">Home</a></li>
                <li><a href="/about">About</a></li>
            </ul>
        </nav>
    </header>
    <main>
        {% block content %}{% endblock %}
    </main>
    <footer>
        <p>&copy; 2025 Python Web Scraping Blog</p>
    </footer>
</body>
</html>

index.html (Homepage Listing All Posts)

{% extends 'base.html' %}

{% block content %}
    <h2>Latest Posts</h2>
    <ul>
        {% for post in posts %}
            <li><a href="/post/{{ post.id }}">{{ post.title }}</a></li>
        {% endfor %}
    </ul>
{% endblock %}

post.html (Individual Post)

{% extends 'base.html' %}

{% block content %}
    <article>
        <h2>{{ post.title }}</h2>
        <p><strong>Published on:</strong> {{ post.created_at }}</p>
        <div>
            {{ post.content | safe }}
        </div>
    </article>
{% endblock %}

Sponsor Key-Word
"This Content Sponsored by Buymote Shopping app
BuyMote E-Shopping Application is One of the Online Shopping App
Now Available on Play Store & App Store (Buymote E-Shopping)
Click Below Link and Install Application: https://buymote.shop/links/0f5993744a9213079a6b53e8
Sponsor Content: #buymote #buymoteeshopping #buymoteonline #buymoteshopping #buymoteapplication"

Step 7: Add Some Example Blog Posts to Database

You can manually add posts to the database using the Flask shell or by creating a script to insert data.

Example of adding blog posts using Python:

from app import app
from models import db, BlogPost

with app.app_context():
    post1 = BlogPost(title="Introduction to Web Scraping", content="Web scraping is...")
    post2 = BlogPost(title="Libraries for Python Web Scraping", content="The best libraries for web scraping are...")
    db.session.add(post1)
    db.session.add(post2)
    db.session.commit()

Step 8: Add Styles (style.css)

Create the CSS file in static/css/style.css.

body {
    font-family: Arial, sans-serif;
    margin: 0;
    padding: 0;
    background-color: #f4f4f4;
}

header {
    background-color: #333;
    color: white;
    padding: 10px;
}

header h1 {
    margin: 0;
    font-size: 2em;
}

nav ul {
    list-style: none;
    padding: 0;
}

nav ul li {
    display: inline;
    margin-right: 15px;
}

nav ul li a {
    color: white;
    text-decoration: none;
}

main {
    padding: 20px;
}

footer {
    background-color: #333;
    color: white;
    text-align: center;
    padding: 10px;
    position: fixed;
    bottom: 0;
    width: 100%;
}

Step 9: Run the Application

Now that everything is set up, you can run your Flask app.

python app.py

This will start the web server at http://127.0.0.1:5000/. You can view your blog homepage, read individual posts, and add new content.

Step 10: Expand the Blog with Topics

To organize the blog by topics such as Python for Web Scraping, you can add a category field to the BlogPost model.

Update models.py:

class BlogPost(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(200), nullable=False)
    content = db.Column(db.Text, nullable=False)
    category = db.Column(db.String(50), nullable=False)  # Added category
    created_at = db.Column(db.DateTime, default=db.func.now())

    def __repr__(self):
        return f"<BlogPost {self.title}>"

Update the index.html to filter by category or display them as tags:

<h2>Blog Topics</h2>
<ul>
    <li><a href="/category/python-web-scraping">Python for Web Scraping</a></li>
    <li><a href="/category/beautifulsoup">BeautifulSoup</a></li>
    <li><a href="/category/scrapy">Scrapy</a></li>
</ul>

You can also create a route to filter posts by category:

@app.route('/category/<category_name>')
def category(category_name):
    posts = BlogPost.query.filter_by(category=category_name).all()
    return render_template('index.html', posts=posts)

Sponsor Key-Word
"This Content Sponsored by Buymote Shopping app
BuyMote E-Shopping Application is One of the Online Shopping App
Now Available on Play Store & App Store (Buymote E-Shopping)
Click Below Link and Install Application: https://buymote.shop/links/0f5993744a9213079a6b53e8
Sponsor Content: #buymote #buymoteeshopping #buymoteonline #buymoteshopping #buymoteapplication"

Step 11: Add User Authentication (Login/Signup)

To allow users to register and log in, we'll use Flask-Login for session management and Flask-WTF for handling forms securely.

11.1: Install Dependencies

First, install the necessary libraries for user authentication and form handling.

pip install flask-login flask-wtf flask-bcrypt

11.2: Update models.py for User Model

We’ll add a User model to handle user accounts.

from flask_sqlalchemy import SQLAlchemy
from datetime import datetime
from flask_bcrypt import Bcrypt

db = SQLAlchemy()
bcrypt = Bcrypt()

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(120), unique=True, nullable=False)
    email = db.Column(db.String(120), unique=True, nullable=False)
    password = db.Column(db.String(200), nullable=False)

    def __repr__(self):
        return f"<User {self.username}>"
    
    def check_password(self, password):
        return bcrypt.check_password_hash(self.password, password)

class BlogPost(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(200), nullable=False)
    content = db.Column(db.Text, nullable=False)
    category = db.Column(db.String(50), nullable=False)
    created_at = db.Column(db.DateTime, default=db.func.now())
    user_id = db.Column(db.Integer, db.ForeignKey('user.id'), nullable=False)

    user = db.relationship('User', backref=db.backref('posts', lazy=True))

    def __repr__(self):
        return f"<BlogPost {self.title}>"

  • User model has username, email, and password.

  • BlogPost model now has a user_id to associate each post with a user.

11.3: Add User Authentication Routes

Update your app.py to manage user registration and login:

from flask import Flask, render_template, redirect, url_for, flash, request
from flask_login import LoginManager, UserMixin, login_user, login_required, logout_user, current_user
from models import db, User, BlogPost
from flask_bcrypt import Bcrypt
from flask_wtf import FlaskForm
from wtforms import StringField, PasswordField, SubmitField
from wtforms.validators import DataRequired, Email, EqualTo

app = Flask(__name__)
app.config['SECRET_KEY'] = 'mysecretkey'
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///blog.db'
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db.init_app(app)
bcrypt = Bcrypt(app)

login_manager = LoginManager()
login_manager.init_app(app)
login_manager.login_view = 'login'


# User Loader
@login_manager.user_loader
def load_user(user_id):
    return User.query.get(int(user_id))


# Login Form
class LoginForm(FlaskForm):
    username = StringField('Username', validators=[DataRequired()])
    password = PasswordField('Password', validators=[DataRequired()])
    submit = SubmitField('Login')


# Register Form
class RegisterForm(FlaskForm):
    username = StringField('Username', validators=[DataRequired()])
    email = StringField('Email', validators=[DataRequired(), Email()])
    password = PasswordField('Password', validators=[DataRequired()])
    confirm_password = PasswordField('Confirm Password', validators=[DataRequired(), EqualTo('password')])
    submit = SubmitField('Register')


# Routes

@app.route('/register', methods=['GET', 'POST'])
def register():
    form = RegisterForm()
    if form.validate_on_submit():
        hashed_password = bcrypt.generate_password_hash(form.password.data).decode('utf-8')
        new_user = User(username=form.username.data, email=form.email.data, password=hashed_password)
        db.session.add(new_user)
        db.session.commit()
        flash('Your account has been created!', 'success')
        return redirect(url_for('login'))
    return render_template('register.html', form=form)


@app.route('/login', methods=['GET', 'POST'])
def login():
    form = LoginForm()
    if form.validate_on_submit():
        user = User.query.filter_by(username=form.username.data).first()
        if user and bcrypt.check_password_hash(user.password, form.password.data):
            login_user(user)
            flash('Login successful!', 'success')
            return redirect(url_for('index'))
        else:
            flash('Login unsuccessful. Check username and password', 'danger')
    return render_template('login.html', form=form)


@app.route('/logout')
@login_required
def logout():
    logout_user()
    return redirect(url_for('index'))


# Other routes (index, post, etc.) go here as before

  • User model: Includes fields for username, email, and password.

  • Flask-WTF forms: Used to create secure forms for login and registration.

  • Flask-Login: Handles login, logout, and session management.

11.4: Create the Login and Register Templates

Create login.html and register.html inside the templates folder.

login.html:

{% extends 'base.html' %}

{% block content %}
    <h2>Login</h2>
    <form method="POST">
        {{ form.hidden_tag() }}
        <div>
            {{ form.username.label }}: {{ form.username() }}
        </div>
        <div>
            {{ form.password.label }}: {{ form.password() }}
        </div>
        <div>
            {{ form.submit() }}
        </div>
    </form>
{% endblock %}

register.html:

{% extends 'base.html' %}

{% block content %}
    <h2>Register</h2>
    <form method="POST">
        {{ form.hidden_tag() }}
        <div>
            {{ form.username.label }}: {{ form.username() }}
        </div>
        <div>
            {{ form.email.label }}: {{ form.email() }}
        </div>
        <div>
            {{ form.password.label }}: {{ form.password() }}
        </div>
        <div>
            {{ form.confirm_password.label }}: {{ form.confirm_password() }}
        </div>
        <div>
            {{ form.submit() }}
        </div>
    </form>
{% endblock %}

11.5: Protect Routes with login_required

Update your blog post creation routes to only allow logged-in users to add new posts.

@app.route('/create', methods=['GET', 'POST'])
@login_required
def create_post():
    if request.method == 'POST':
        title = request.form['title']
        content = request.form['content']
        category = request.form['category']
        new_post = BlogPost(title=title, content=content, category=category, user_id=current_user.id)
        db.session.add(new_post)
        db.session.commit()
        flash('Your post has been created!', 'success')
        return redirect(url_for('index'))
    return render_template('create_post.html')

Sponsor Key-Word
"This Content Sponsored by Buymote Shopping app
BuyMote E-Shopping Application is One of the Online Shopping App
Now Available on Play Store & App Store (Buymote E-Shopping)
Click Below Link and Install Application: https://buymote.shop/links/0f5993744a9213079a6b53e8
Sponsor Content: #buymote #buymoteeshopping #buymoteonline #buymoteshopping #buymoteapplication"

Step 12: Create Post Submission Form

We’ll also need a form to submit new blog posts. Create a create_post.html template for this.

{% extends 'base.html' %}

{% block content %}
    <h2>Create New Post</h2>
    <form method="POST">
        <div>
            <label>Title</label>
            <input type="text" name="title" required>
        </div>
        <div>
            <label>Content</label>
            <textarea name="content" required></textarea>
        </div>
        <div>
            <label>Category</label>
            <input type="text" name="category" required>
        </div>
        <div>
            <button type="submit">Create Post</button>
        </div>
    </form>
{% endblock %}

Step 13: Implement User-Specific Content

To make the website even more user-specific, you can show the logged-in user's posts on the homepage or provide an option for them to edit or delete their posts.

Show User's Posts on Dashboard

@app.route('/dashboard')
@login_required
def dashboard():
    posts = BlogPost.query.filter_by(user_id=current_user.id).all()
    return render_template('dashboard.html', posts=posts)

Create dashboard.html

{% extends 'base.html' %}

{% block content %}
    <h2>Your Posts</h2>
    <ul>
        {% for post in posts %}
            <li>{{ post.title }} - <a href="/post/{{ post.id }}">Read</a></li>
        {% endfor %}
    </ul>
{% endblock %}

Conclusion

At this point, you've built a blog website with user authentication, the ability to create posts, and the framework for handling user-specific content. Here are some further improvements you could add next:

  1. Post Editing and Deletion: Allow users to edit or delete their posts.

  2. Markdown Support: Let users write posts in Markdown format.

  3. Pagination: Implement pagination for the blog posts.

  4. Admin Panel: Allow admins to manage all posts

Sponsor Key-Word

"This Content Sponsored by Buymote Shopping app

BuyMote E-Shopping Application is One of the Online Shopping App

Now Available on Play Store & App Store (Buymote E-Shopping)

Click Below Link and Install Application: https://buymote.shop/links/0f5993744a9213079a6b53e8

Sponsor Content: #buymote #buymoteeshopping #buymoteonline #buymoteshopping #buymoteapplication"

By using Flask, Flask-Login, Flask-WTF, and Flask-SQLAlchemy, you now have a solid foundation for a Python web scraping blog website with authentication and post creation functionality. Keep building and expanding!

1. Homepage (/):

  • The homepage displays a list of blog posts on the topic of Python Web Scraping.

  • Each blog post will show its title as a link, which leads to the individual post page.

  • Visitors can browse posts on various topics (e.g., BeautifulSoup, Scrapy, Selenium) organized in a list.

Example Output (Homepage):

Python Web Scraping Blog
-------------------------------------------
Home | About

Latest Posts
-------------------------------------------
- Introduction to Web Scraping
- Libraries for Python Web Scraping
- Handling Real-World Challenges in Web Scraping

2. Individual Blog Post Page (/post/<id>):

  • When users click on a blog post title from the homepage, they are directed to the individual post page.

  • The individual post page displays the post's full content and a timestamp for when it was created.

Example Output (Post Page):

Introduction to Web Scraping
-------------------------------------------
Published on: 2025-07-09 14:32:45

Web scraping is the process of extracting data from websites...

3. User Authentication:

Login Page (/login):

  • Users can log in using their username and password.

  • On successful login, they are redirected to the homepage or their dashboard (if they have one).

Example Output (Login Page):

Login
-------------------------------------------
Username: [_______]
Password: [_______]
[Login]

Register Page (/register):

  • New users can register by providing a username, email, password, and confirm password.

Example Output (Register Page):

Register
-------------------------------------------
Username: [_______]
Email:    [_______]
Password: [_______]
Confirm Password: [_______]
[Register]

4. Dashboard (/dashboard):

  • After logging in, users can view their personal dashboard, which lists all the posts they've created.

  • Each post can be linked to a view page where the user can edit or delete their posts.

Example Output (Dashboard):

Your Posts
-------------------------------------------
- Introduction to Web Scraping - [Read]
- Libraries for Python Web Scraping - [Read]

5. Create New Post Page (/create):

  • Registered and logged-in users can create new blog posts from the "Create New Post" page.

  • They can add title, content, and category to their post and then submit it to the database.

Example Output (Create Post Page):

Create New Post
-------------------------------------------
Title: [_________]
Content: [_________]
Category: [_________]
[Create Post]

6. Post Management (Edit/Delete):

  • If you're expanding the project with features to edit or delete posts, users can modify or remove posts they have created from the dashboard or the post's detail page.

7. User Logout:

  • A logged-in user can log out, which will end the current session and redirect them back to the homepage.

Example Output (Logout):

  • Users will be redirected to the homepage with a message such as "You have been logged out successfully."

Final Output Flow

  1. Homepage (/):

    • Displays a list of all blog posts.

  2. Login/Register:

    • Users can log in to create, edit, or delete posts.

  3. Dashboard (/dashboard):

    • Logged-in users can see and manage their posts.

  4. Create Post (/create):

    • Registered users can add new blog posts.

  5. Post Page (/post/<id>):

    • Displays the full content of a blog post when clicked.

  6. Logout (/logout):

    • Logs out the user and redirects to the homepage.

Sponsor Key-Word

"This Content Sponsored by Buymote Shopping app

BuyMote E-Shopping Application is One of the Online Shopping App

Now Available on Play Store & App Store (Buymote E-Shopping)

Click Below Link and Install Application: https://buymote.shop/links/0f5993744a9213079a6b53e8

Sponsor Content: #buymote #buymoteeshopping #buymoteonline #buymoteshopping #buymoteapplication"

Example of a Full Blog Cycle (for a User)

  1. First Visit:

    • User lands on the homepage showing a list of blog posts.

  2. Login:

    • The user navigates to the login page, enters credentials, and logs in.

  3. Dashboard:

    • After logging in, the user is redirected to their dashboard, where they can see the list of their previously created posts.

  4. Create New Post:

    • The user clicks on Create Post, enters the title, content, and category for the new post, and submits it.

  5. Post View:

    • The post appears on the homepage. The user can click the title to view the full post.

  6. Logout:

    • After managing the posts, the user logs out, and the session is terminated.

Possible Enhancements:

  1. Admin Panel:

    • If you want to manage content from an admin perspective, you can add an admin user who has more control (e.g., delete any post, approve user registrations).

  2. Pagination:

    • If you have a large number of blog posts, you can implement pagination to load a certain number of posts per page.

  3. Comment System:

    • Allow users to comment on each post.

  4. Markdown Support:

    • Enable users to write posts using Markdown, which can then be converted to HTML for display.

  5. Styling Enhancements:

    • Improve the website's look by using CSS frameworks like Bootstrap or custom styling for a more polished and professional appearance.

Screenshot or Mockup:

If you were to create a mockup, here’s what it could look like:

Homepage:

+-----------------------------------------+
|       Python Web Scraping Blog         |
|-----------------------------------------|
| Home | About | Login                   |
+-----------------------------------------+

Latest Posts:
-----------------------------------------
| - Introduction to Web Scraping         |
| - Libraries for Python Web Scraping    |
| - Handling Real-World Challenges      |
+-----------------------------------------+

Post Page:

+-----------------------------------------+
|     Introduction to Web Scraping       |
|-----------------------------------------|
| Published on: 2025-07-09 14:32:45       |
|-----------------------------------------|
| Content:                                 |
| Web scraping is the process of...       |
+-----------------------------------------+

Dashboard:

+-----------------------------------------+
|      Your Posts                        |
|-----------------------------------------|
| - Introduction to Web Scraping         |
| - Libraries for Python Web Scraping    |
+-----------------------------------------+

This simple blog website should work well for users to read and interact with posts, manage their own content, and have a user-friendly experience.

Comments

Post a Comment